For the user, operating systems often provide an abstraction in the form of a black box that hides not only what is going on on the hardware side (until a non-recoverable error shows up...), but also in the internals of the OS itself. However, system administrators and developers who need to find and fix problems with functional as well as non-functional properties of a system need a better insight into the details of the currently running system. Here, problems with functional properties including things that fail (an operation returns an error and we need to know the reason why), whereas non-functional problems imply that the system fulfills its service, but not as good as expected -- e.g., the system might run too slow or consume too much energy.
In the last two decades, tools were developed that provide a better insight into system details by allowing introspection at runtime, often being able to cross the borders between kernel, runtime libraries and high-level code, e.g. written in Python. One of the first tools to enable introspection is DTrace [1], available on Sun Solaris, MacOS X and FreeBSD (there's also a Linux port). DTrace allows to add code to retrieve values from a running system using a special scripting language, which is compiled to safe bytecodes executed inside the OS kernel.
More recent developments, such as the Linux eBPF system [2], also allow the implementation of certain system functionality [3]. This makes the system much more flexible, but also dangerous to use. Thus, static analysis methods try to provide ways to ensure safe execution of eBPF code [4]. A list of interesting links related to eBPF can be found at [5].
This topic shows that it is possible to access information and even change behavior inside the OS black box. Introspection approaches enable the user to analyze a system using a vertical approach, going from a problem showing up in a program written in a scripting languge through the machine code of the system libraries and the kernel down to the device level.