Non-Euclidean geometry is just one example of a Riemannian space with constant curvature. In differential geometry we have tools for studying all possible geometries. You have learned about non-Euclidean geometry and got impressed. But you did not learn Riemannian geometry and Kaluza-Klein theories. Non-Euclidean geometry was a tip of an iceberg, and the whole iceberg had been discovered later.And that brings me back to non-Euclidean Geometry. What if there are TWO (or more) mathematics with different laws that apply in different circumstances AND/OR different realms? Isn’t this a huge problem? Falling prey to looking for the one-size-fits-all solution?
The same with information theory. We now have a tip of an iceberg, and the whole iceberg is being slowly and painfully discovered now, by many researchers working in this direction. The hidden dimensions of information are still hidden from us today. But they will come out.
Back to your non-Euclidean geometry: this is not another kind of mathematics. It is the same old mathematics, only more advanced. Needs more calculus (covariant derivative) and more algebra (tensors). See for instance: Non-Euclidean geometries here