can’t see the forest through the trees

:on the kernel | linux vs. unix | operating system vs. ubuntu vs. virtual machine | shell vs. bash | command line vs. terminal

As daunting as I imagined the task of writing this article would be, it wasn’t worse than the feeling I continually face of having to learn to program software without a solid grasp of the meaning of the aforementioned components and applications. I often feel confused as to the differences between each of the above listed components and systems. My main problem is that I miss the big picture of how my computer software that I utilize is structured and connected to the rest of my computer. This also creates problems for how I communicate about what I’ve coded on my computer, especially for people that use different software and different machines. Not being able to explain how I’ve been using a computer is at least unflattering and certainly a weak foundation for becoming a Software Engineer. Thus, I began researching for this post and my hope is that others may benefit from my organization.

Instead of focusing on the distinctions of each of the concepts, I simply attempt to explain a basic understandings of these concepts, and move forward from there.  Some helpful resources that have helped with my understanding of these concepts have been Wikipedia, and Black Duck Open Hub.

Each section attempts to provide clarification on these main categories: general term for a kind of software component, system, or application or brand name for a licensed software product, aliases, languages written in, examples, licenses.

kernel

general term for a kind of software component, system, or application
aliases: None yet
written in: C, C++, Objective C, Assembly
examples: Linux Kernel, Hyru Hybrid Kernel, Monolithic kernel vs. Microkernel.
license: N/A

The kernel is a computer program that is the core of a computer's operating system, with complete control over everything in the system. It is the first program loaded on start-up and also handles the rest of start-up, input/output requests from software and translating requests into data-processing instructions for the central processing unit of the computer hardware.  The kernel handles memory and other hardware peripherals like keyboards, monitors, printers, speakers. The kernel has full access to the system's memory and must allow processes to safely access this memory as they require it. The kernel's method of allocating memory allows every program to function properly and thus prevents applications from crashing each other. The kernel's primary function is to mediate application's and other software system's access to the computer's hardware and resources, including: (1) The central processing unit, (2) Random-access memory (RAM), (3) Input/output (I/O) devices.  Essentially, the kernel is the connection from software to hardware.

Monolithic Kernel vs. Microkernel

A well-known debate on the topic of Kernels is the Tanenbaum-Torvalds Debate on Microkernel (Minix) vs. Monolithic Kernel (Linux). Essentially, Tenenbaum wrote in 1991 that microkernel architecture is a better design principle, likely to be the future of kernels, and is ideal in critical situations where reliability is of uttermost importance. Linus Torvalds's response in 1991 was the defense that his monolithic design is superior because of its multithreading abilities and simple design with the trade-off of not being very portable. Briefly, a monolithic kernel is an OS architecture where the entire operating system (which includes the device drivers, file system, and the application IPC) is working in kernel space. Monolithic kernels are able to dynamically load (and unload) executable modules at runtime. With a microkernel architecture, the core functionality is isolated from system services and device drivers (which are basically just system services).  In the below image, in each of the examples, the Red represents the Kernel, Yellow is user mode. The operating system is below the solid horizontal black line along with all the processes that occur within the kernel, and system processes are above the horizontal line, showing briefly the methods that applications interact with the kernel.

kernel structure

operating system

general term for a kind of software component, system, or application
aliases: OS
written in: C, C++, C#, Objective C, Assembly
examples: Windows (based on IBM DOS), Mac OS (based on BSD Unix OS), Ubuntu (based on Linux), Debian (based on Linux), MINIX, CP/M (Control Program/ Monitor), Seattle DOS (Disk Operating System or Quick & Dirty Operating System QDOS)
license: N/A

Your computer's operating system (OS) is software that sits underneath all of the other software on a computer, receiving requests from those programs and relaying these requests to the computer’s hardware; it's role is of a manager for all of the software and hardware on the computer.  The kernel is a part of the Operating System, but the shell is separate because it is an application that is running on the operating system.  Most of the time, there are several different computer programs running at the same time, and they all need to access your computer's central processing unit (CPU), memory, and storage. The operating system coordinates all of this to make sure each program gets what it needs.  For large systems, the operating system has even greater responsibilities and power for sure that different program's and user's tasks running at the same time do not interfere with each other. The operating system is also responsible for security (i.e. ensuring that unauthorized users do not access the system) and memory management.  Most operating systems come with a variety of built-in applications such as: a terminal emulator, text editors, advanced GUI document and spreadsheet editors, internet browsers, calculators, calendars, games, and image viewers, media players, and other various and popular editors.

Components: kernel, program execution, interrupts, modes, memory management, virtual memory, multitasking, disk access & file systems, device drivers, security, networking, user interface.

linux

brand name for a licensed software product
aliases: alias linux='echo -e ${PATH//:/n}'
written in: C
examples: Ubuntu, Fedora, Kubuntu, WinLinux, Parsix, Pinguy
license: GNU.org

Linux was created in 1991 by Linus Torvalds, while a student at the University of Helsinki. Torvalds originally built Linux as a free and open source alternative to Minix, which was primarily used in academic settings. For more on the differences between linux and minix, please check out the above discussion in the kernel section on microkernels vs. monolithic kernels. According to opensource.com:

Linus originally intended to name it “Freax,” but the administrator of the server Torvalds used to distribute the original code named his directory “Linux” after a combination of Torvalds’ first name and the word Unix, and the name stuck.

In an interview with TED.com, Linus explains that his original intention was not to use open source methodology to improve the Linux software, but rather he made the source code publicly available to view because he wanted to show off his work and see what feedback he would receive from others.  Then, a friend at the University of Helsinki encouraged him to make it open-source, and despite his worries of others taking his work to benefit instead of him, he took a risk and made Linux open-source.  At that point in his life, Linus explained in the referenced TED interview that he was very pleased and impressed by the feedback and suggestions that he received since most of his life he had primarily viewed his code from his own unique perspective.  This helped to inspire the growth of Linux and the amount of contributors to Linux software, which today has over 1000 engineers contributing to the Linux Kernel alone.  Linus explains that the paradox of Linux and other popular open-source technologies that he has pioneered is that they connect people and other people's ideas, yet they were designed to allow Linus to remain introverted and disconnected from people.  Since this article has not yet discussed GNU open-source software, now is a good time to provide some reference to what open-source is because Linux is the best-known and most-used open source operating system.

GNU Open Source Software

The GNU General Public License (GNU GPL or GPL) is a free software license, which gives users the freedom to run, study, share and modify software with GNU public license. The license was originally written by Richard Stallman of the Free Software Foundation (FSF) for the GNU Project. GNU stood for: GNU's Not Unix, which is a recursive acronym. There are many other Open-source and general public licenses, but this article focuses on software in the GNU collection, which is one of the most popular software licenses in the free and open-source software domain. Popular software under the GNU license: Bash, Gimp, Git, Linux, Emacs, GNU Compiler Collection and GNU C Library. Linux software slowly grew to have many contributors with the open-source license, and so Linus needed a way to better manage all the developers and their code contributions, which is what inspired him to develope git technology.

Git

Git software is also free, open-source, and is the most widely used modern version control system in the world; it connects multiple engineers to be able to contribute to the same projects at a very organized and fast pace.  Git has become popularized by github.com, which utilizes git software to host shared projects for easy access to the latest version of a software project.

linux kernel

unix

brand name for a licensed software product
aliases: BSD, FreeBSD, Sun, OpenBSD, AIX, Unix Ware, Solaris, Xenix
written in: Haskell, C, Autoconf
examples: unix is an operating system; originally released in 1973
license: The Open Group

The original Unix operating system was developed at AT&T's Bell Labs research center in 1969. In the 1970s and 1980s, AT&T licensed Unix to third-party vendors, leading to the development of several Unix variants, including Berkeley Unix, HP-UX, AIX, and Microsoft's Xenix. In 1993, AT&T sold the rights to the Unix operating system to Novell, Inc., which a few years later sold the Unix trademark to the consortium that eventually became The Open Group.

unix distribution tree

ubuntu

brand name for a licensed software product
aliases: Ubuntu is a Nguni Bantu term meaning "humanity", and often understood to mean 'the belief in a universal bond of sharing that connects all humanity.'
written in: C, C#, C++, Autoconf, Assembly, Python, Perl, Bash, Java, and many more.
examples: Various computer languages can be coded, compiled and executed in Ubuntu including: C, C++, Java, Fortran, Python, Perl, PHP, Ruby, TCL, Lisp, and Bash.
license: No License, but it's components have various licenses usually under GNU.org.

Ubuntu is a distribution of Linux, which means that it is a version of the operating system that has the Linux kernel as its foundation along with various other software packages included with the Ubuntu distribution. There are approximately over 500 different Linux distributions in use; many are free and have communities of users who provide each other with guidance and support, but Ubuntu is unique for its extensive community and professional support and maintenance. With automobiles as an analogy, Linux is like the engine under the hood of a car. The entire car is like the Ubuntu package with various components and software packages, Linux being the most important as it composes the entire engine. A typical Linux distribution comprises a Linux kernel, GNU tools and libraries, additional software, documentation, a window system, window manager, and a desktop environment. Most of the included software is open-source software (explanation above on GNU open-source software).

virtual machine

general term for a kind of software component, system, or application
aliases: VM
written in: Not written yet
examples: Virtual Box, Parallels, QEMU, VMware, Windows Virtual PC, Bochs, Xen, Bootcamp.
License: MIT License

A virtual machine is a software application consisting of multiple files and components that together have the power of an entire physical computer system; it can powers up, boot, shuts down, reboot, run an operating system and run other applications. The virtual machine (also known as the guest) is comprised of a set of specification and configuration files and is backed by the physical resources of the computer it is running on (also known as the host). Every virtual machine has virtual devices that provide the same functionality as physical hardware and have additional benefits in terms of portability, manageability, and security. Virtual machines are more common than is often realized. Many small businesses and individuals utilize VPS hosting for their website packages. VPS stands for "Virtual Private Server", which is another name for a "virtual machine". Large companies with large and powerful servers (i.e computers) are able to create multiple virtual machines for clients to be able to host their own websites. Some of the advantages of a virtual machine include: (1) It allows multiple operating system environments on a single physical computer. This means that your Mac or PC can use a virtual machine to run any operating system including a linux based OS, Windows, or Mac OS. (2) Virtual machines are widely available and are easy to manage and maintain. (3) VM's offer application provisioning and disaster recovery options. For example, if you accidentally damage the system you are building, it can be easily deleted and recovered from a backup so long as you have a working computer. Some of the negatives of virtual machines include: (1) They are not as efficient as a physical computer because the hardware resources are distributed in an indirect way. (2) Multiple VMs running on a single physical machine can deliver unstable and slow performance. Therefore, in order to run a virtual machine, it is recommended that your computer have enough RAM to be able to accommodate the RAM needs of both machines running on your computer hardware.

shell

general term for a kind of software component, system, or application
aliases: command-line interpreter
written in: C, Fortran, Assembly, .NET
examples: sh (Bourne Shell), bash (Bourne-Again SHell), zsh, csh, fish, (friendly interactive shell), MS-DOS, cmd.exe (Command Prompt), PowerShell
license: N/A

A shell is an application that is run with the operating system, but is separate from the operating system.  The shell is a program that accepts commands as text input and converts commands to appropriate operating system functions.  A shell is a user interface that is used for access to an operating system's services; also known as a command line interpreter.  A command line interpreter like a shell, relies on either a command-line interface (CLI or a terminal emulator) or graphical user interface (GUI) to take input commands from the user and implement the commands through running other applications or outputting the appropriate responses.  It is named a shell because it is a layer around the operating system's kernel.  The shell is distinct from the kernel because the user interacts with the shell, then the shell gives instructions to the kernel which interacts with hardware and other applications to implement the user requests.  Wikipedia has a complex comparison of many different shell systems here: Comparison of Command Shells. Additionally, GNU.org has a helpful description of the shell in the following excerpt:

At its base, a shell is simply a macro processor that executes commands. The term macro processor means functionality where text and symbols are expanded to create larger expressions. A shell allows execution of GNU commands.  Shells also provide a small set of built-in commands (builtins) implementing functionality impossible or inconvenient to obtain via separate utilities.  While executing commands is essential, most of the power (and complexity) of shells is due to their embedded programming languages.

bash

brand name for a licensed software product
aliases: Bourne Again Shell
written in: C
examples: bash is commonly used with GNU Linux and Mac OS with terminal emulators
license: GNU.org Open Source

Straight outta the horses mouth at gnu.org:

Bash is the shell, or command language interpreter, for the GNU operating system. The name is an acronym for the ‘Bourne-Again SHell’, a pun on Stephen Bourne, the author of the direct ancestor of the current Unix shell sh, which appeared in the Seventh Edition Bell Labs Research version of Unix.

Bash is the default shell for Linux distributions and for mac OS, it is used to execute commands read from the standard input or from a file.

console or text based user interface

general term for a kind of software component, system, or application
aliases: tty (teletype), text input/output environment, terminal, Basic Input Output System (BIOS), command line interface
written in: C, Objective C, C++, Assembly
examples: Synchronet, Midnight Commander, FreeDos, IBM DOS, XFdrake, Corvus Concept, VT100
license: N/A

Digital Equipment Corporation's VT100

The image here is of the Digital Equipment Corporation's VT100, one of the first terminals to support ANSI escape codes for cursor control and other tasks, and also had a number of extended codes for special features like controlling the status lights on the keyboard. The console is the text entry and display device for system administration messages, particularly those from theBasic Input Output System BIOS or boot loader, the kernel, from the init system and from the system logger. It is a physical device consisting of a keyboard and a screen, and traditionally is a text terminal, but may also be a graphical terminal.  The device is used to interact with the kernel, which has various functions described above, and the console also displays output based upon the instructions of the user input or running application directions.

terminal emulator

general term for a kind of software component, system, or application
aliases: Terminal Window, Terminal, Command Prompt, command line interface, text input/output environment, Basic Input Output System (BIOS)
written in: C, Objective C, C++, Assembly
examples: Command Prompt (aka cmd.exe) used with Windows, Terminal used on Mac OS, xterm, iterm, xterm, terminator, rxvt, GNOME terminal
license: N/A, depends on the software

A terminal emulators or command line interfaces are closely similar to the "terminal" or "console" described above.  Most current personal computers use what is known as a graphical user interface or GUI, which adds visual enhancements to understand the organization of the computer and graphics and images to help with remembering all the commands that a computer can interpret.  However, many programmers want access to a more complete set of commands, the ability to combine commands or even code new commands, and so GUI's have an application known as a terminal emulator to replicate the ancestor of GUI's which was the text based user interface described above.  The main distinction between the terminal and a terminal emulator, is that the emulator acts as a middle step between the user and the shell by interpreting user input for the shell to receive and interpreting output from the shell for the user to read.  Older terminals and consoles did not have this extra step.  In mac OS, the emulator program is called terminal. Windows typically uses the "command prompt" or cmd.exe.