Access Control Lists on Linux Explained

Access Control Lists on Linux Explained | Linux ACL Cheat Sheet

Access control list (ACL) gives an additional, more flexible permission mechanism for file systems. It is intended to help UNIX file permissions. ACL permits you to grant permissions for any user or group to any disc resource.

If you are working as a system administrator then you would probably be familiar with Linux ACLs. Because they were used to define more fine-grained discretionary access rights for files and directories.

In today’s Access Control Lists on Linux Explained Tutorial, we are going to explain deeper information about Linux access control lists, what they are used for and how they are managed to configure a Linux system properly.

Get Ready to Learn A New Topic?

What You Will Learn

If you follow this tutorial until the end, you are going to learn about the following topics:

That’s quite a long program, so without further ado, let’s start with a quick definition of what Linux file access control lists acls are.

Access Control Lists Basics on Linux

On Linux, there are two ways of setting permissions for users and groups: with regular file permissions or with access control lists.

What is ACL(Access Control Lists)?

Access control lists are used on Linux filesystems to set custom and more personalized permissions on files and folders. ACLs allow file owners or privileged users to grant rights to specific users or to specific groups.

getfacl-1

In Linux, as you probably know, the permissions are divided into three categories: one for the owner of the file, one for the group, and one for the others.

However, in some cases, you may want to grant access to a directory (the execute permission for example) to a specific user without having to put this user into the group of the file.

This is exactly why access control lists were invented in the first place.

Do Refer More Linux Tutorials: 

Listing Access Control List

On Linux, access control lists are not enabled when you create a new file or directory on your host (except if a parent directory has some ACLs predefined).

To see if access control lists are defined for a file or directory, run the ls command and look for a “+” character at the end of the permission line.

$ ls -l

access-control-list
To show the difference, here is the difference when listing files on a minimal instance.

Listing Access Control List acl-fileNow that you have some basics about access control lists, let’s see how you can start creating basic ACL for your files and directories.

List of commands for setting up ACL

1) To add permission for user
setfacl -m "u:user:permissions" /path/to/file

2) To add permissions for a group
setfacl -m "g:group:permissions" /path/to/file

3) To allow all files or directories to inherit ACL entries from the directory it is within
setfacl -dm "entry" /path/to/dir

4) To remove a specific entry
setfacl -x "entry" /path/to/file

5) To remove all entries
setfacl -b path/to/file

Creating access control lists on Linux

Before starting with ACL commands, it is important to have the packages installed on your host.

Checking ACL packages installation

It might not be the case if you chose to have a minimal server running.

Start by checking the help related to the setfacl by running the following command

$ setfacl --help

If your host cannot find the setfacl command, make sure to install the necessary packages for ACL management.

$ sudo apt-get install acl -y

Note that you will need sudo privileges on Debian 10 to run this command.

Checking ACL packages installation

Run the setfacl command and make sure that you are able to see the help commands this time.

Now that your host is correctly configured, let’s see how the setfacl command works.

Setting access control lists using setfacl

With access control lists, there are two main commands that you need to remember: setfacl and getfacl.

In this chapter, we are going to take a look at the setfacl command as the getfacl one is pretty self-explanatory.

The setfacl command is used on Linux to create, modify and remove access control lists on a file or directory.

The setfacl has the following syntax

$ setfacl {-m, -x}  {u, g}:<name>:[r, w, x] <file, directory>

Where curly brackets mean one of the following options and regular brackets mean one or several items.

  • -m: means that you want to modify one or several ACL entries on the file or directory.
  • -x: means that you want to remove one or several ACL entries on a file or directory.
  • {u, g}: if you want to modify the ACL for a user or for a group.
  • name: this is an optional parameter, it can be omitted if you want to set the ACL entries for every user or for every group on your host.
  • [r, w, x]: in order to set read, write or execute permissions on the file or directory.

For example, in order to set specific write permissions for a user on a file, you would write the following command

$ setfacl -m u:user:w <file, directory>

In order to set execute permissions for all users on your host, you would write the following command

$ setfacl -m u::x <file, directory>

To set full permissions for a specific group on your host, you would write the setfacl this way

$ setfacl -m g:group:rwx <file, directory>

Now let’s say that you want to remove an ACL entry from a file.

In order to remove a user-specific entry from a file, you would specify the x option.

Note: you cannot specific rights from a single ACL entry, meaning that you can’t remove write permissions, keeping the ACL read permissions active.

$ setfacl -x u:<user> <file, directory>

Similarly, to remove ACL related to groups on your host, you would write the following command

$ setfacl -x g:<group> <file, directory>

Now that you have seen how you can create access control lists easily on Linux, it is time to see how you can check existing access control lists on files and directories.

Listing access control lists using getfacl

The getfacl command is used on Linux to print a complete listing of all regular permissions and access control lists permissions on a file or directory.

The getfacl can be used with the following syntax

$ getfacl <file, directory>

getfacl-2

The getfacl command is divided into multiple categories :

  • Filename, owner, and group: The information about the user and group ownership is shown at the top;
  • User permissions: First, you would find regular user permissions, also called the owning user, followed by any user-specific ACL entries (called named users)
  • Group permissions: Owning groups are presented followed by group-specific ACL entries, also called named groups
  • Mask: That restricts the permissions given to ACL entries, the mask is going to be detailed in the next section;
  • Other permissions: Those permissions are always active and this is the last category explored when no other permissions match with the current user or group.

Working with the access control lists mask

As you probably saw from the last screenshot, there is a mask entry between the named groups and the other permissions.

But what is this mask used for?

The ACL mask is different from the file creation mask (umask) and it is used in order to restrict existing ACL entries existing on a file or directory.

The ACL mask is used as the maximum set of ACL permissions regardless of existing permissions that exceed the ACL mask.

As always, a diagram speaks a hundred words.

Working with the access control lists mask

The ACL mask is updated every time you run a setfacl command unless you specify that you don’t want to update the mask with the -n flag.

To prevent the mask from being updated, run the setfacl with the following command

$ setfacl -n -m u:antoine:rwx <file, directory>

As you can see in this example, I have set the user “antoine” to have full permissions on the file.

The mask is set to restrict permissions to read and write permissions.

As a consequence, the “effective permissions” set on this file for this user are read and write ones, the execute permission is not granted.

Working with the access control lists mask

Note: If your maximum set of permissions differs from the mask entry, you will be presented with an effective line computing the “real” set of ACL entries used.

Creating access control lists defaults on directories

As already mentioned in this article, it is possible to create ACL entries on directories and they work in the same way file access control lists work.

However, there is a small difference when it comes to directories: you have to option to create access control lists defaults.

Access control lists defaults are used to create ACL entries on a directory that will be inherited by objects in this directory like files or subdirectories.

When creating default ACL entries :

  • Files created in this directory inherit the ACL entries specified in the parent directory
  • Subdirectories created in this directory inherit the ACL entries as well as the default ACL entries from the parent directory.

To create default ACL entries, specify the -d option when setting ACL using the setfacl command.

$ setfacl -d -m {u, g}:<name>:[r, w, x] <directory>

For example, to assign read permissions to all files created in a directory, you would run the following command

$ setfacl -d -m u::r directory

Creating access control lists defaults on directories getfacl-3

Now, when a file is created in this acl-directory, you can see that default ACL entries are applied to the file.

Creating access control lists defaults on directories default-1-1

Similarly, when a directory is created in the acl-directory, it will inherit default ACL entries specified in the parent directory.

Creating access control lists defaults on directories default-2

Note that it is recommended to specify default permissions for all three categories (user, group, and other).

In fact, specifying one of the three entries will create the remaining two with permissions related to the file creation mask.

Deleting default access control lists on directories

In order to delete default existing access control lists on directories, use the -k flag with the setfacl command.

$ setfacl -k <directory>

Given the example we specified earlier, here is how to delete default entries

$ setfacl -k acl-directory

Deleting default access control lists on directories remove-default

Note that deleting ACL entries from the parent directory does not delete ACL entries in files or directories contained in the parent directory.

To remove default ACL entries in a directory and all subdirectories, you would have to use a recursive option (-R)

$ setfacl -kR <directory>

remove-default-1

Conclusion

In this tutorial, you learned about access control lists on Linux, the getfacl, and the setfacl command.

You also discovered more about the access control lists mask and how default ACL is used in order to create ACL entries on files and subdirectories contained in the parent directory.

If you are curious about Linux system administration, we have many more tutorials on the subject, make sure to read them!

Understanding Hard and Soft Links on Linux

Understanding Hard and Soft Links on Linux | What are Hard & Soft Links in Linux?

In this tutorial, we are going to discuss what are hard and soft links with syntax and how we can understand Hard and Soft Links on Linux easily.

In case, you are wondering how you can generate a shortcut on a Linux system, then this tutorial can be the perfect answer for you all.

Are you Ready to Start learning about Understanding Hard and Soft Links on Linux? here you go.

What Will You Learn?

This section is completely about the stuff that is provided in this tutorial about the topic we are going to discuss. It helps you to know a bit earlier about what you are going to learn:

  • How storage works on a Linux system and what inodes are exactly?
  • What hard and soft links are, given what you just learned before
  • How copying differs from creating links to your files
  • How to create links on a Linux system
  • How to find hard and soft links: all the commands that you should know.
  • Some of the quick facts about hard and soft links

That’s a long program, so without further ado, let’s see how data is organized on your Linux system and what inodes are?

Do Refer: 

How does storage work on a Linux system?

In order to understand what hard and soft links are, you need to have some basics on how data is stored on your system.

In a computer, data are stored on a hard drive.

Your hard drive has a capacity, let’s say 1 TB, and it is divided into multiple blocks of a given capacity.

II - How does storage work on a Linux system storage-design

If I launch a simple fdisk command on my system, I am able to see that my device is separated into sectors of 512 bytes.

II - How does storage work on a Linux system fdisk-simple

That’s a lot of sectors, as my hard drive has a capacity of almost 32 GBs.

Now every time that I create a file, it will be stored on a block.

But what if the file size exceeds 512 bytes?

You guessed it, my file will be “fragmented” and stored into multiple different blocks.

how-files-are-stored

If the different pieces of your file are physically far away from each other on the disk, the time needed to build your file will obviously be longer.

That’s why you had to defragment your disk on Windows systems, for example, you were essentially making related blocks closer.

Luckily for us, those blocks have addresses.

Your Linux system does not have to read the entire disk to find your file, it will cherry-pick some addresses that correspond to your file data.

How?

By using inodes.

a – What are inodes?

Inodes are essentially identification cards for your file.

They contain the file metadata, the file permissions, the file type, the file size but most importantly the physical address of this file on the disk.

inode

Without going into too many details, your inode will keep references to the locations of your different file blocks.

So one inode equals one file.

That’s the first layer of your file system, and how references are kept to the underlying data.

storage-first-layer

However, as you probably noticed, filenames are not part of the inode, they are not part of the identification card of the file.

b – About filenames and inodes

On a Linux system, filenames are kept on a separate index.

This separate index keeps track of all the different filenames existing on your system (even directories) and they know the corresponding inode in the inode index.

What does it mean?

It essentially means that you can have multiple filenames (say “doc.txt” and “paper.txt”) pointing to the same exact file, sharing the same content.

Now that you learned about the filenames index, let’s add another layer to our previous layered architecture.

b – About filenames and inodes filesystems-design

With this schema, you have a basic understanding of how files are organized and stored on your filesystem, but more importantly, you will be able to understand what hard and soft links are.

What is Soft Link And Hard Link In Linux?

Let’s start with softs links as they are probably the easiest to understand.

a – Understanding soft links

Soft links, also called symbolic links, are files that point to other files on the filesystem.

Similar to shortcuts on Windows or macOS, soft links are often used as faster ways to access files located in another part of the filesystem (because the path may be hard to remember for example).

Symbolic links are identified with the filetype “l” when running a ls command.

They also have a special syntax composed of the link name and an arrow pointing to the file they are referencing.

a – Understanding soft links symbolic-link

In this case, the file “shortcut” is pointing to the file “file.txt” on my filesystem.

Have you paid attention to the permissions?

They are set to “rwx” by default for the user, the group, and the others.

However, you would be constrained by the permissions of the file if you were to manipulate this file with another user.

b – Soft links and inodes

So why did we talk so much about inodes in the first section?

Let’s have a quick look at the inode of the file and the inode of the shortcut.

$ stat shortcut
  File: shortcut -> file.txt
  Size: 3               Blocks: 0          IO Block: 4096   symbolic link
Device: fc01h/64513d    Inode: 258539      Links: 1

$ stat file.txt
  File: job
  Size: 59              Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 258545      Links: 2

The inodes are different.

However, the original file inode is pointing directly to the file content (i.e the blocks containing the actual content) while the symbolic link inode is pointing to a block containing the path to the original file.

sooft-links

The file and the shortcut share the same content.

It means that I was to modify the content of the shortcut, the changes would be passed on to the content of the file.

If I delete the shortcut, it will simply delete the reference to the first inode. As the file inode (the first one) still references the content of the file on the disk, the content won’t be lost.

However, if I were to delete the file, the symbolic link would lose its reference to the first inode. As a consequence, you would not be able to read the file anymore.

This is what we call a dangling symbolic link, a link that is not pointing to anything.

deleting-soft-link-original-file

See the red highlighting when I deleted the original file?

Your terminal is giving visual clues that a symbolic link is a dangling symbolic link.

So what’s the size of a symbolic link?

Remember, the symbolic link points to the path of the original file on the filesystem.

In this example, I created a file named “devconnected”, and I built a shortcut to it using the ln command.

Can you guess the size of the shortcut? 12, because “devconnected” actually contains 12 letters.

soft-link-size

Great!

Now you have a good understanding of what soft links are.

c – Understanding hard links

Hard links to a file are instances of the file under a different name on the filesystem.

Hard links are literally the file, meaning that they share all the attributes of the original file, even the inode number.

hard-link

Here’s a hard link created on my system.

It is sharing the same content as the original file and it is also sharing the same permissions.

Changing permissions of the original file would change the permissions of the hard link.

d – Hard links and inodes

Let’s have a quick look at the original file inode and the hard link inode.

$ stat hardlink
  File: hardlink
  Size: 59              Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 258545      Links: 2

$ stat file.txt
  File: file.txt
  Size: 59              Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 258545      Links: 2

As you probably noticed, the inodes are the same, but the filenames are different!

Here’s what happens in this case on the filesystem.

hard-soft-links

When you are creating a symbolic link, you are essentially creating a new link to the same content, via another inode, but you don’t have access to the content directly in the link.

When creating a hard link, you are literally directly manipulating the file.

If you modify the content in the hard link file, the content will be changed in the original file.

Similarly, if you modify the content in the original file, it will be modified in the hard link file.

However, if you delete the hard link file, you will still be able to access the original file content.

Similarly, deleting the original file has no consequences on the content of the hard link.

Data are definitely deleted when no inodes point to it anymore.

Hard or Soft?

You won’t find a clear answer to this question. If the type that suits your special situation can be the best link. While these concepts can be tricky to memorize, the syntax is pretty straightforward, so that is a plus!

To keep the two links easily separated in your mind, I leave you with this:

  • A hard link always points a filename to data on a storage device.
  • A soft link always points a filename to another filename, which then points to information on a storage device.

What is the difference between copying and creating a hard link?

With the concepts that you just learned, you may wonder what’s the difference between copying a file and creating a hard link to the file.

Don’t we have two files in the end, with the same content?

The difference between copying and hard linking is that hard-linking does not duplicate the content of the file that it links to.

When you are copying a file, you are essentially assigning new blocks on the disk with the same content as the original file.

Even if you share the same content with hard-linking, you are using disk space to store the name of the original file, not the actual content of the file.

Diagrams may explain it better than words.

Here’s what copying a file means.

copying-a-file

See how it differs from hard-linking?

Now that you know how copying files differ from hard linking to files, let’s see how you can create symbolic and hard links on a Linux system.

Manipulating links on a Linux system

a – How to create a symbolic link on Linux?

To create a symbolic link, you need to use the ln command, with a -s flag (for symbolic).

The first argument specifies the file you want to link to.

The second argument describes the name of the link you are about to create.

$ ln -s file shortcut

$ ls -l
-rw-rw-r-- 1 schkn schkn 0 Aug 14 20:12 file
lrwxrwxrwx 1 schkn schkn 4 Aug 14 20:12 shortcut -> file

You can also create symbolic links to directories.

$ mkdir folder
$ ln -s folder shortcut-folder

$ ls -l
drwxrwxr-x  2 schkn schkn   4096 Aug 14 20:13 folder
lrwxrwxrwx  1 schkn schkn      7 Aug 14 20:14 shortcut-folder -> folder/

b – How to delete symbolic links on Linux

To remove existing symbolic links, use the unlink command.

Following our previous example :

$ unlink shortcut

$ ls -l 
-rw-rw-r-- 1 schkn schkn 0 Aug 14 20:12 file

You can also simply remove the shortcut by using the rm command.

Using the unlink command might be a little bit safer than performing an rm command.

$ rm shortcut

$ ls -l 
-rw-rw-r-- 1 schkn schkn 0 Aug 14 20:12 file

c – How to create a hard link on Linux

A hard link is created using the lnwithout specifying the s flag.

$ ln file hardlink

$ ls -l
-rw-rw-r--  2 schkn schkn      0 Aug 14 20:12 file
-rw-rw-r--  2 schkn schkn      0 Aug 14 20:12 hardlink

d – How to remove a hard link on Linux

Again, you can use the unlink command to delete a hard link on a Linux system.

$ ln file hardlink
$ unlink hardlink
$ ls -l
-rw-rw-r--  2 schkn schkn      0 Aug 14 20:12 file

Now that you know how to create links, let’s see how you can find links on your filesystem.

How to find links on a filesystem?

There are multiple ways to find links on a Linux system, but here are the main ones.

Using the find command

The find command has a type flag that you can use in order to find links on your system.

$ find . -type l -ls
262558      0 lrwxrwxrwx   1 schkn    schkn           7 Aug 14 20:14 ./shortcut-folder2 -> folder2/
.
.
.
262558      0 lrwxrwxrwx   1 schkn    schkn           7 Aug 14 20:14 ./shortcut-folder -> folder/

However, if you want to limit searches to the current directory, you have to use the maxdepth parameter.

$ find . -maxdepth 1 -type l -ls 
262558      0 lrwxrwxrwx   1 schkn    schkn           7 Aug 14 20:14 ./shortcut-folder -> folder/
258539      0 lrwxrwxrwx   1 schkn    schkn           3 Jan 26  2019 ./soft-job -> job

Finding links that point to a specific file

With the lname option, you have the opportunity to target links pointing to a specific filename.

$ ls -l
drwxrwxr-x  2 schkn schkn   4096 Aug 14 20:13 folder
lrwxrwxrwx  1 schkn schkn      7 Aug 14 20:38 devconnected -> folder/

$ find . -lname "fold*"
./devconnected

Finding broken links

With the L flag, you can find broken (or daggling) symbolic links on your system.

$ find -L . -type l -ls
258539      0 lrwxrwxrwx   1 schkn    schkn           3 Jan 26  2019 ./broken-link -> file

Quick facts about links on Linux

Before finishing this tutorial, there are some quick facts that you need to know about soft and hard links.

  • Soft links can point to different filesystems, and to remote filesystems. If you were to use NFS, which stands for Network File System, you would be able to create a symbolic link from one filesystem to a file system accessed by the network. As Linux abstracts different filesystems by using a virtual filesystem, it makes no difference for the kernel to link to a file located on an ext2, ext3, or an ext4 filesystem.
  • Hard links cannot be created for directories and they are constrained to the limits of your current filesystem. Creating hard links for directories could create access loops, where you would try to access a directory that points to itself. If you need more explanations about why it can’t be done conceptually, here’s a very good post by Volker Siegel on the subject.

I hope that you learned something new today. If you are looking for more Linux system administration tutorials, make sure to check our dedicated section.

Here is the list of our recent tutorials:

Cron Jobs and Crontab on Linux Explained

Cron Jobs and Crontab on Linux Explained | What is Cron Job & Crontab in Linux with Syntax?

This Cron Jobs and Crontab on Linux Explained Tutorial helps you understand cron on Linux along with the role of the crontab file. System administrators are likely to spend a lot of time performing recurring tasks on their systems.

But the best way to automate tasks on Linux systems is the cron jobs. It was initially found in 1975 by AT&T Bell Laboratories.

Not only cron jobs and crontab command on Linux but also you are going to explain about Linux cron daemon. Rather than these concepts also keep focusing on the difference between user-defined cron jobs and system-defined cron jobs.

Are you ready for learnings?

What is ‘crontab’ in Linux?

The crontab is a list of commands that you require to run on a daily schedule, and also the name of the command used to manage that list. Crontab stands for “cron table,” because it uses the job scheduler cron to execute tasks.

cron itself is termed after “Chronos, ” the Greek word for time.cron is the system process that will automatically execute tasks for you as per the set schedule. The schedule is called the crontab, which is also the name of the program used to edit that schedule.

Linux Crontab Format

MIN HOUR DOM MON DOW CMD

Linux Crontab Syntax

Field    Description    Allowed Value
MIN      Minute field    0 to 59
HOUR     Hour field      0 to 23
DOM      Day of Month    1-31
MON      Month field     1-12
DOW      Day Of Week     0-6
CMD      Command         Any command to be executed.

Important Crontab Examples

The following are some of the necessary examples of Crontab. Kindly have a look at them:

Description Command
Cron command to do the various scheduling jobs. The below-given command executes at 7 AM and 5 PM daily.
0 7,17 * * * /scripts/script.sh
Command to execute a cron after every 5 minutes.
*/5* * * * *  /scripts/script.sh
Cron scheduler command helps you to execute the task every Monday at 5 AM. This command is helpful for doing weekly tasks like system clean-up.
0 5 * * mon  /scripts/script.sh
Command run your script at 3 minutes intervals.
*/3 * * * * /scripts/monitor.sh

Linux Crontab Command

The crontabcommand permits you to install, view, or open a crontab file for editing:

  • crontab -e: Edit crontab file, or create one if it doesn’t already exist.
  • crontab -l: Display crontab file contents.
  • crontab -r: Remove your current crontab file.
  • crontab -i: Remove your current crontab file with a prompt before removal.
  • crontab -u <username>: Edit other user crontab files. This option needs system administrator privileges.

What is Cron and Cron Jobs in Linux?

Cron is a system daemon run on any Linux system that is responsible for detecting cron jobs and executing them at given intervals.

Cron runs every minute and it will inspect a set of pre-defined directories on your filesystem to see if jobs need to be run.

On the other hand, cron jobs are tasks defined to run at given intervals or periods, usually shell scripts or simple bash commands.

Cron jobs are usually used in order to log certain events to your Syslog utilities, or to schedule backup operations on your host (like database backups or filesystem backups).

For a Linux OS running systemd as a service manager, you can inspect the cron service by running the following command

$ sudo systemctl status cron.service

Note: You need sudo privileges to inspect system services with systemd

What is the Cron Job Syntax?

The most important to know about cron is probably the cron job syntax.

In order to define a cron job, you are going to define:

  • Periodicity: meaning when your job is going to be executed over time. You can define it to run every first day of the month, every 5 minutes, or on a very specific day of the year. Examples will be given later on in the article;
  • Command: literally the command to be executed by the cron utility, it can be a backup command or whatever command that you would normally run in a shell;
  • User: this is reserved for system-defined cron jobs where you want to specify the user that should the cron command. For user-defined cron jobs, you don’t have to specify a user, and your system will run them as root by default.

cron-syntax

As you probably noticed, the periodicity column is made of 5 columns.

Every single column can either be set to *, meaning that the command will be executed for every single value of the interval specified or to a particular value, for example, the 6th month of the year.

If you want to execute a command for every minute, of every hour, of every day of the month, of every month, you need to write the following command

* * * * *  logger "This is a command executed every minute"

If you want to execute a command every 30 minutes, you would write

*/30 * * * * logger "This is executed every 30 minutes"

On the other hand, if you want to execute a command on the first day of the month at midnight, you would write

0 0 1 * * logger "This is executed on the first day of the month, at midnight"

When defining those cron jobs, I didn’t have to specify the user executing them.

This is because there is a difference between user-defined cron jobs and system-defined cron jobs.

User-Defined Cron Jobs

User-defined cron jobs are cron jobs defined by a given user on the host. It doesn’t mean that it is not able to execute commands affecting the entire system, but its tasks are isolated on given folders on the host.

Every user is able to have its own set of cron jobs on a Linux host.

Listing user-defined cron jobs

When connected to a specific user, run this command to see the cron jobs owned by the user

$ crontab -l

If you own cron jobs, they will immediately be displayed to the standard output.

By default, user-defined cron jobs are stored in the /var/spool/cron/crontabs directory but you will need to be root to explore it.

user-defined-cron

Adding user-defined cron jobs

In order to edit the cron jobs related to the user you are connected to, run the following command

$ crontab -e

By default, your host will open your default editor and you will be able to able your cron jobs on the system.

Add a new line to the end of the file with the following line for example

* * * * * logger "This is a log command from junosnotes"

Logger is a command that allows users to write custom messages to logs. If you need a complete guide on logging and Syslog, we have a complete write-up on it.

You don’t have to specify a user as the system is already aware of the user defining this command.

Moreover, the command will be executed as the current user by default.

You don’t have to restart any services, your job will be taken into account on the next occurrence.

Given the example, we specified earlier, inspect your logs to see your cron job executed

$ sudo journalctl -xfn

cron-job-user

As you can see, the cron service inspected user-specific directories on the host (in /var/spool/cron/crontabs), it opened a session as my current user, executed the command, and closed the session.

Awesome!

You learned how you can define user-defined cron jobs on your host.

Removing user defined cron jobs

In order to remove user-defined cron jobs, use the following command

$ crontab -r
(or)
$ crontab -ri

Crontab will be deleted for your current user (it won’t delete system-defined cron jobs).

Run a cron job listing to check that all cron jobs have been deleted

System Defined Cron Jobs

System-defined cron jobs are jobs defined in shared directories on the filesystem.

It means that, given that you have sudo privileges on the host, you will be able to define cron jobs that may be modified by other administrators on your system.

Directories related to system defined cron jobs are located in the etc directory and can be seen by running

$ ls -l | grep cron

As you can see, this folder contains a lot of different folders and files :

  • anacrontab: a file used by the anacron service on Linux, which will be explained in one of the next sections.
  • cron.d: a directory containing a list of cron jobs to be read by the cron service. The files in cron.d are written given the cron syntax we saw before;
  • cron.daily: a directory containing a list of scripts to be executed by the system every day. Files are different from the files contained in the cron.d directory because they are actual bash scripts and not cron jobs written with cron syntax;
  • cron.hourly, cron.monthly, cron.weekly are self-explanatory, they contain scripts executed every hour, every month, and every week of the year;
  • crontab: a cron file written with cron syntax that instructs the cron service to run jobs located in the daily, hourly, monthly, and weekly folders. It can also define custom jobs similarly to user-defined cron jobs, except that you have to specify the user that should run the command.

Listing system defined cron jobs

As you probably understood, cron jobs defined in global configuration folders are spread over multiple folders.

Moreover, multiple cron files can be defined on those folders.

However, using the command line, there are efficient ways to concatenate all the files in a given directory.

To list all cron jobs defined in cron.d, run the following command

$ cat /etc/cron.d/*

Similarly, to list cron jobs defined in the crontab file, run the following command

$ cat /etc/crontab

Similarly, you can inspect all the scripts that are going to be executed on a daily basis

ls -l /etc/cron.daily/

Adding system defined cron jobs

As you probably understood, there are multiple ways to add system-defined cron jobs.

You can create a cron file in the cron.d, and the file will be inspected every minute for changes.

You can also add your cron job directly to the crontab file. If you want to execute a task every minute or every hour, it is recommended to add your script directly to the corresponding cron directories.

The only difference with user-defined cron jobs is that you will have to specify a user that will run the cron command.

For example, create a new file in the cron.d directory and add the following content to it (you will obviously need sudo privileges for the commands to run)

$ sudo nano /etc/cron.d/custom-cron

*/1 * * * *    root    logger 'This is a cron from cron.d'

Again, no need for you to restart any services, the cron service will inspect your file on the next iteration.

To see your cron job in action, run the following command

$ sudo journalctl -xfn 100 | grep logger

This is what you should see on your screen

Great!

As you can see your job is now executed every minute by the root user on your host.

Now that you have a complete idea of what user-defined cron jobs and system-defined cron jobs are, let’s see the complete cron lifecycle on a Linux host.

Cron Complete Cycle on Linux

Without further ado, here is a complete cron cycle on Linux.

cron-cycle-2

This is what your cron service does every minute, as well as all the directories inspected.

Cron will inspect the user-defined cron jobs and execute them if needed.

It will also inspect the crontab file where several default cron jobs are defined by default.

Those default cron jobs are scripts that instruct your host to verify every minute, every hour, every day, and every week specific folders and to execute the scripts that are inside them.

Finally, the cron.d directory is inspected. The cron.d may contain custom cron files and it also contains a very important file which is the anacron cron file.

Anacron cron file on Linux

The anacron cron file is a file executed every half an hour between 7:00 am and 11 pm.

The anacron cron file is responsible for calling the anacron service.

The anacron service is a service that is responsible for running cron jobs in case your computer was not able to run them in the first place.

Suppose that your computer is off but you had a cron job responsible for running update scripts every week.

As a consequence, when turning your computer on, instead of waiting an entire week to run those update scripts, the anacron service will detect that you were not able to launch the update cron before.

Anacron will then proceed to run the cron job for your system to be updated.

By default, the anacron cron file is instructed to verify that the cron.daily, cron.weekly, cron.hourly, and cron.monthly directories were correctly called in the past.

If anacron detects that the cron jobs in the cron.monthly folders haven’t run, it will be in charge to run them.

Conclusion

Today, you learned how cron and crontab work on Linux. You also had a complete introduction on the cron syntax and how to define your own cron scripts as a user on your host.

Finally, you had a complete cron cycle overview of how things work on your host and of what anacron is.

If you are interested in Linux system administration, we have a complete section about it on our website. Check out some of our Linux Tutorials from below:

Until then, have fun, as always.

How To Encrypt Partition on Linux

In one of our previous articles, we learnt how you can encrypt your entire root filesystem on Linux easily.

However, in some cases, you may want to encrypt one simple partition that may store some of your important files.

As you already know, encrypting your disks is crucial. If your laptop were to be stolen, you would probably lose all your personal information.

However, there are some ways for you to cope with this problem : by encrypting your disk partitions.

In this tutorial, you will learn about all the steps necessary to encrypt an entire disk partition, secure it with a passphrase or with a keyfile.

For the example, the article will be illustrated on a RHEL 8 operating system, but there should not be any differences if you use another one.

Prerequisites

In order to execute most of the commands provided in this article, you need to have administrator rights.

To check whether this is the case or not, you can execute the “groups” command and verify that you belong to the “sudo” group for Debian-based distributions or “wheel” for RedHat based ones.

$ groups

How To Encrypt Partition on Linux groups

If you don’t have such rights, you can read one of our articles on the subject about getting sudo rights for Ubuntu or CentOS distributions.

Encrypt Partition using cryptsetup

As specified in the previous articles, encrypting a partition involves formatting the entire disk.

As a consequence, if you plan on encrypting a disk with existing data, you should now that your data will be erased in the process. To avoid losing anything, you should make a backup of your data on an external disk or in an online cloud.

Create New Partition on disk

In order to encrypt a partition, we are going first to create a new one using the “fdisk” utility. For the example, we are going to create a new partition named “sdb1” on the “sdb” disk.

$ sudo fdisk /dev/sdb

Create New Partition on disk fdisk-utility

In the fdisk utility, you can create a new partition using the “n” keyword and specify that you want a partition with a size of 5 GBs for example.

Create New Partition on disk create-partition-fdisk

If you are not sure about how to use “fdisk” or how to create partitions, we have a dedicated article about this subject.

At the end of the process, you need to use the “w” keyword in order to write the changes to the disk.

Create New Partition on disk write-changes-disk

Awesome, now that your partition is created, we are going to format it as a LUKS partition.

Format Disk Partition as LUKS

To encrypt the partition, we are going to use a command related to the LUKS project.

The LUKS project, short for Linux Unified Key System, is a specification used in order to encrypt all storage devices using special cryptographic protocols. As described, LUKS is only a specification, you will need a program that implements it.

In this case, we are going to use the “cryptsetup” utility. As explained in the manual section, cryptsetup aims at creating encrypted spaces for dm-crypt.

First of all, make sure that you have the “cryptsetup” command using the “which” command.

$ which cryptsetup

Format Disk Partition as LUKS which-cryptsetup

If the cryptsetup cannot be found on your server, make sure to install it using one of the following commands

$ sudo apt-get install cryptsetup                     (for Debian distributions)

$ sudo yum install cryptsetup                         (for RHEL/CentOS distributions)

To create your LUKS partition, you need to execute the “cryptsetup” followed by “luksFormat” and the name of the partition to be formatted.

$ sudo cryptsetup luksFormat /dev/sdb1

Format Disk Partition as LUKS cryptsetup-luksformat

First of all, you are reminded that encrypting your disk will actually format it in the process.

After typing “YES” in capital letters, you will have to choose a passphrase in order to secure your device.

LUKS supports two ways of protecting your media : using a passphrase (the one that we currently use) and using keys. For now, you can choose a safe password and your partition should be formatted automatically.

Now that your partition is created, you can inspect it using the “lsblk” command : the partition should be marked as “crypto_luks“.

$ lsblk -f

lsblk-command

Awesome! Now that the volume is formatted, we can open it and create a simple ext4 filesystem on it.

Create ext4 Filesystem on Partition

By default, your encrypted volume is closed meaning that you cannot access data that is available on it.

In order to “open”, meaning “unlocking” your volume, you have to use the “cryptsetup” command again followed by “luksOpen” and the name of the volume.

At the end of the command, provide a name for your open volume, in this case we are going to choose “cryptpart“.

$ sudo cryptsetup luksOpen /dev/sdb1 cryptpart

Create ext4 Filesystem on Partition cryptsetup-luksopen

As you can guess, you are asked to provide the passphrase that you chose in the previous section.

Running the “lsblk” command again, you probably noticed that one volume was created under the “sdb1” encrypted volume named “cryptpart“. The “device mapper”, which is one of the frameworks of the Linux Kernel, did that for you.

Now that your volume is unlocked, it is time for you to create a new ext4 filesystem on it.

To create a new filesystem on your partition, use the “mkfs” command followed by the filesystem format, in this case “ext4”.

$ sudo mkfs.ext4 /dev/mapper/cryptpart

mkfs-command

Awesome, the filesystem was created.

You can now mount it and add new files to it. Files created on this volume will automatically be encrypted.

$ mkdir -p /home/devconnected/files 

$ sudo mount /dev/mapper/cryptpart /home/devconnected/files

$ sudo chown devconnected:devconnected /home/devconnected/files

mount-encrypted-partition

Awesome, now that your data is safe on an encrypted partition, let’s see how you can mount the encryption partition on boot.

Modify crypttab and fstab files

Many system administrators know the existence of the fstab file that is used by your init process to mount drives.

However, when dealing with encrypted partitions, there is another file that comes into play : /etc/crypttab.

Similarly to the fstab file, crypttab is read by your init process when booting. Given the information provided in it, it will ask you to unlock the partition or it will read a key file in order to do it automatically.

Note : the /etc/crypttab may not exist on your system. If it is not the case, you may have to create it.

crypttab-columns

The columns of the crypttab are described above :

  • Device name : you can give your decrypted device any name that you want. Furthermore, it will be automatically created by the device mapper under the “/dev/mapper” path. In the previous section, we chose “cryptpart” for this column;
  • Encrypted device UUID : in order to find which partition contains the encrypted data, your system needs to have its UUID meaning its unique identifier;
  • Method of authentication : as explained, you can choose “none” for the passphrase or you can specify a path to the key. The key method will be explained in the last chapter of this article;
  • Mount options : using this column, you can specify the number of tries for a passphrase, the cipher, the encryption method and many other parameters. The complete list of options is available in the “crypttab” manual page.
$ sudo nano /etc/crypttab

# Content of the crypttab file
cryptpart    UUID=<partition_uuid>    none    luks

crypttab-file-2

If you have doubts about the UUID of your encrypted partition, you can use the “blkid” command with a simple “grep” pipe.

$ sudo blkid | grep -i luks

blkid-command

Now that the “/etc/crypttab” file is modified, you will have to modify the “fstab” file to specify the mountpoint.

$ sudo blkid | grep -i ext4

$ sudo nano /etc/fstab

fstab-file

In the fstab columns, you have to specify :

  • The decrypted device UUID : in order to find it, you can use the “blkid” command but make sure that you opened the device before proceeding. If the device is closed, you won’t be able to find your UUID;
  • The mount point : where the decrypted device is going to be mounted. If the path does not exist, it is going to be created automatically;
  • The filesystem type : in this case, we chose to use “ext4” but it may be different on your system;
  • Dump and pass options : we don’t want the filesystem to be checked on boot-time, so we can keep it to the default values.

When you are done, save your file and you should be good to go.

Given the steps you just performed, your device is ready and it should automatically be mounted on boot.

Verify encrypted device mounting on boot

In order to verify that the device is correctly mounted, we can restart our server and wait for the initramfs module to open the encrypted device.

$ sudo reboot

Verify encrypted device mounting on boot encryption-boot

This is the screen that you should see, at least on RHEL8, when starting your server. If you provide the passphrase, your machine should be able to unlock it and mount it for you.

Once you are logged in your server, you can check that the encrypted partition was correctly mounted using the “lsblk” once again.

$ lsblk -f | grep sdb1 -A 2

lsblk-mounted

Congratulations, you successfully encrypted a partition on Linux using LUKS!

Create Keys For Encrypted Partition

As explained before, LUKS handles two authentication methods, namely passphrases and key files.

In the previous section, we used passphrases but it can be quite handy for you to also have a authentication key.

First of all, create a key file and store it somewhere safe (in directories that regular users cannot navigate to, like “/boot” or “/root“).

$ echo "supersecretpass" > volume-key

$ sudo mv volume-key /boot/

create-volume-key

As you can see, by default, the file was created using the user credentials and it has too many permissions.

Using the “chown” and “chmod” commands, we can set “root” as the owner of the file and change its permissions to read-only.

$ sudo chown root:root /boot/volume-key

$ sudo chmod 0400 /boot/volume-key

read-only-file

Now that the file is set to read-only, we can add it as a key in one of the slots of our LUKS volume.

Add Key to LUKS Volume

In order to add a key to your LUKS volume, you need to execute the “cryptsetup” command followed by the “luksAddKey”, the name of the encrypted volume and the path to the key.

$ sudo cryptsetup luksAddKey <encrypted_device> <path_to_key>

$ sudo cryptsetup luksAddKey /dev/sdb1 /boot/volume-key

Add Key to LUKS Volume luks-add-key

In order to perform this operation, you will be prompted for your passphrase. When provided, the key will be automatically added to your keyslots.

To verify that the key was correctly added, you can inspect your keyslots using the “luksDump” command.

$ sudo cryptsetup luksDump /dev/sdb1

Add Key to LUKS Volume luks-dump-command

Now that the key is added, you only need to modify the “/etc/crypttab” in order for your system to find it on boot.

$ sudo nano /etc/crypttab

# Content of the crypttab file
cryptpart    UUID=<partition_uuid>    /boot/volume-key    luks

When rebooting, your encrypted partition will be mounted automatically!

auto-mount-linux

Conclusion

In this article, you learnt how you can easily encrypt your partition on Linux using the LUKS project and its implementation named cryptsetup.

You can saw that you can use a “key file” in order for your partition to be unlocked automatically.

If you are interested in a full system encryption, we recently wrote an article on the subject.

Also, if you want to read more about Linux System Administration, make sure to have a look at our dedicated section on the website.

How To Flush DNS Cache on Linux

DNS, short for the Domain Name System protocol, is used on Linux systems in order to retrieve IP addresses associated with names.

For example, when you are performing a ping request, it is quite likely that you are using the DNS protocol to retrieve the server IP.

In most cases, the DNS requests that you perform are stored in a local cache on your operating system.

However, in some cases, you may want to flush the DNS cache of your server.

It might be because you changed the IP of a server on your network and you want to changes to be reflected immediately.

In this tutorial, you are going to learn how you can easily flush the DNS cache on Linux, whether you are using systemd or dnsmasq.

Prerequisites

In order to be able to flush your DNS cache, you have to know how DNS resolution works on your Linux system.

Depending on your distribution, you may be facing different Linux services that act as a DNS resolver.

Before you start, it is quite important for you to know how DNS resolution will actually happen on your operating system.

How To Flush DNS Cache on Linux dns-resolution-linux
Inspired by this Wikipedia diagram
If you are reading this article, you are looking to flush the cache of your local DNS resolver. But as you can see, there are many different caches from your local application until the actual Internet DNS servers.

In this tutorial, we are going to focus on the yellow box meaning the local stub resolver implemented on every Linux system.

Finding your local DNS resolver

On most Linux systems, the DNS resolver is either “systemd-resolved” or dnsmasq. In order to know if you are dealing with one or another, you can execute the following command

$ sudo lsof -i :53 -S
Note : so why are we running this command? As DNS runs on port 53, we are looking for the commands associated with the service running on port 53, which is your local DNS resolver or “stub”.

Finding your local DNS resolver lsof-command
As you can see, on a recent Ubuntu 20.04 distribution, the service listening on port 53 is systemd-resolved. However, if you were to execute this command on Ubuntu 14.04, you would get a different output.
lsof-command-old-distribution

In this case, the local DNS used in dnsmasq and commands are obviously different.

local-dns-resolvers

Knowing this information, you can go the chapter you are interested in. If you were to have a different output on your server, make sure to leave a comment for us to update this article.

Flush DNS using systemd-resolved

The easiest way to flush the DNS on Linux, if you are using systemd-resolved, is to use the “systemd-resolve” command followed by “–flush-caches”.

Alternatively, you can use the “resolvectl” command followed by the “flush-caches” option.

$ sudo systemd-resolve --flush-caches

$ sudo resolvectl flush-caches

In order to verify that your Linux DNS cache was actually flushed, you can use the “–statistics” option that will highlight the “Current Cache Size” under the “Cache” section.

$ sudo systemd-resolve --statistics

flush-dns-systemd-resolve

Congratulations, you successfully flushed your DNS cache on Linux!

Flush DNS cache using signals

Another way of flushing the DNS cache can be achieved by sending a “USR2” signal to the “systemd-resolved” service that will instruct it to flush its DNS cache.

$ sudo killall -USR2 systemd-resolved

In order to check that the DNS cache was actually flushed, you can send a “USR1” signal to the systemd-resolved service. This way, it will dump its current state into the systemd journal.

$ sudo killall -USR1 systemd-resolved

$ sudo journalctl -r -u systemd-resolved

Flush DNS cache using signals flush-dns-using-signals

Awesome, your DNS cache was correctly flushed using signals!

Flush DNS using dnsmasq

The easiest way to flush your DNS resolver, when using dnsmasq, is send a “SIGHUP” signal to the “dnsmasq” process with the “killall” command.

$ sudo killall -HUP dnsmasq

Flush DNS using dnsmasq flush-dnsmasq

Similarly to systemd-resolved, you can send a “USR1” to the process in order for it to print its statistics to the “syslog” log file. Using a simple “tail” command, we are able to verify that the DNS cache was actually flushed.

Now what if you were to run dnsmasq as a service?

Dnsmasq running a service

In some cases, you may run “dnsmasq” as a service on your server. In order to check whether this is the case or not, you can run the “systemctl” command or the “service” one if you are on an SysVinit system.

$ sudo systemctl is-active dnsmasq

# On SysVinit systems
$ sudo service dnsmasq status

If you notice that dnsmasq is running as a service, you can restart it using the usual “systemctl” or “service” commands.

$ sudo systemctl restart dnsmasq

# On SysVinit systems
$ sudo service dnsmasq restart

After running those commands, always make sure that your services were correctly restarted.

$ sudo systemctl status dnsmasq

# On SysVinit systems
$ sudo service dnsmasq status

Conclusion

In this tutorial, you learnt how you can quickly and easily flush your DNS cache on Linux.

Using this article, you can easily clear the cache for systemd and dnsmasq local resolvers. However, you should know that there is another common DNS, named bind, that is purposefully omitted in this article.

Another article about setting up a local DNS cache server using BIND should come in the near future.

If you are interested in DNS queries and how they are performed, you can use this very useful article from “zwischenzugs” named the Anatomy of a DNS query. The article is particularly useful if you want to debug DNS queries and you wonder how they are performed.

Also if you are interested in Linux System Administration, we have a complete section about it on the website, so make sure to check it out.

Input Output Redirection on Linux Explained

Input Output Redirection on Linux Explained | Error Redirection in Linux

In Linux, it’s very often to execute Input or output redirection while working daily. It is one of the core concepts of Unix-based systems also it is utilized as a way to improve programmer productivity amazingly.

In this tutorial, we will be discussing in detail regarding the standard input/output redirections on Linux. Mostly, Unix system commands take input from your terminal and send the resultant output back to your terminal.

Moreover, this guide will reflect on the design of the Linux kernel on files as well as the way processes work for having a deep and complete understanding of what input & output redirection is.

Do Check: 

If you follow this Input Output Redirection on Linux Explained Tutorial until the end, then you will be having a good grip on the concepts like What file descriptors are and how they related to standard inputs and outputs, How to check standard inputs and outputs for a given process on Linux, How to redirect standard input and output on Linux, and How to use pipelines to chain inputs and outputs for long commands;

So without further ado, let’s take a look at what file descriptors are and how files are conceptualized by the Linux kernel.

Get Ready?

What is Redirection?

In Linux, Redirection is a feature that helps when executing a command, you can change the standard input/output devices. The basic workflow of any Linux command is that it takes an input and gives an output.

  • The standard input (stdin) device is the keyboard.
  • The standard output (stdout) device is the screen.

With redirection, the standard input/output can be changed.

What are Linux processes?

Before understanding input and output on a Linux system, it is very important to have some basics about what Linux processes are and how they interact with your hardware.

If you are only interested in input and output redirection command lines, you can jump to the next sections. This section is for system administrators willing to go deeper into the subject.

a – How are Linux processes created?

You probably already heard it before, as it is a pretty popular adage, but on Linux, everything is a file.

It means that processes, devices, keyboards, hard drives are represented as files living on the filesystem.

The Linux Kernel may differentiate those files by assigning them a file type (a file, a directory, a soft link, or a socket for example) but they are stored in the same data structure by the Kernel.

As you probably already know, Linux processes are created as forks of existing processes which may be the init process or the systemd process on more recent distributions.

When creating a new process, the Linux Kernel will fork a parent process and it will duplicate a structure which is the following one.

b – How are files stored on Linux?

I believe that a diagram speaks a hundred words, so here is how files are conceptually stored on a Linux system.

b – How are files stored on Linux?

As you can see, for every process created, a new task_struct is created on your Linux host.

This structure holds two references, one for filesystem metadata (called fs) where you can find information such as the filesystem mask for example.

The other one is a structure for files holding what we call file descriptors.

It also contains metadata about the files used by the process but we will focus on file descriptors for this chapter.

In computer science, file descriptors are references to other files that are currently used by the kernel itself.

But what do those files even represent?

c – How are file descriptors used on Linux?

As you probably already know, the kernel acts as an interface between your hardware devices (a screen, a mouse, a CD-ROM, or a keyboard).

It means that your Kernel is able to understand that you want to transfer some files between disks, or that you may want to create a new video on your secondary drive for example.

As a consequence, the Linux Kernel is permanently moving data from input devices (a keyboard for example) to output devices (a hard drive for example).

Using this abstraction, processes are essentially a way to manipulate inputs (as Read operations) to render various outputs (as write operations)

But how do processes know where data should be sent to?

Processes know where data should be sent to using file descriptors.

On Linux, the file descriptor 0 (or fd[0]) is assigned to the standard input.

Similarly the file descriptor 1 (or fd[1]) is assigned to the standard output, and the file descriptor 2 (or fd[2]) is assigned to the standard error.

file-descriptors-Linux

It is a constant on a Linux system, for every process, the first three file descriptors are reserved for standard inputs, outputs, and errors.

Those file descriptors are mapped to devices on your Linux system.

Devices registered when the kernel was instantiated, they can be seen in the /dev directory of your host.

c – How are file descriptors used on Linux?

If you were to take a look at the file descriptors of a given process, let’s say a bash process for example, you can see that file descriptors are essentially soft links to real hardware devices on your host.

c – How are file descriptors used on Linux proc

As you can see, when isolating the file descriptors of my bash process (that has the 5151 PID on my host), I am able to see the devices interacting with my process (or the files opened by the kernel for my process).

In this case, /dev/pts/0 represents a terminal which is a virtual device (or tty) on my virtual filesystem. In simpler terms, it means that my bash instance (running in a Gnome terminal interface) waits for inputs from my keyboard, prints them to the screen, and executes them when asked to.

Now that you have a clearer understanding of file descriptors and how they are used by processes, we are ready to describe how to do input and output redirection on Linux.

What is Output redirection on Linux?

Input and output redirection is a technique used in order to redirect/change standard inputs and outputs, essentially changing where data is read from, or where data is written to.

For example, if I execute a command on my Linux shell, the output might be printed directly to my terminal (a cat command for example).

However, with output redirection, I could choose to store the output of my cat command in a file for long-term storage.

a – How does output redirection works?

Output redirection is the act of redirecting the output of a process to a chosen place like files, databases, terminals, or any devices (or virtual devices) that can be written to.

output-redirection-diagram

As an example, let’s have a look at the echo command.

By default, the echo function will take a string parameter and print it to the default output device.

As a consequence, if you run the echo function in a terminal, the output is going to be printed in the terminal itself.

echo-devconnected

Now let’s say that I want the string to be printed to a file instead, for long-term storage.

To redirect standard output on Linux, you have to use the “>” operator.

As an example, to redirect the standard output of the echo function to a file, you should run

$ echo junosnotes > file

If the file is not existing, it will be created.

Next, you can have a look at the content of the file and see that the “junosnotes” string was correctly printed to it.

redirecting-output-to-a-file

Alternatively, it is possible to redirect the output by using the “1>” syntax.

$ echo test 1> file

a – How does output redirection works

b – Output Redirection to files in a non-destructive way

When redirecting the standard output to a file, you probably noticed that it erases the existing content of the file.

Sometimes, it can be quite problematic as you would want to keep the existing content of the file, and just append some changes to the end of the file.

To append content to a file using output redirection, use the “>>” operator rather than the “>” operator.

Given the example we just used before, let’s add a second line to our existing file.

$ echo a second line >> file

appending-content-output-redirection

Great!

As you can see, the content was appended to the file, rather than overwriting it completely.

c – Output redirection gotchas

When dealing with output redirection, you might be tempted to execute a command to a file only to redirect the output to the same file.

Redirecting to the same file

echo 'This a cool butterfly' > file
sed 's/butterfly/parrot/g' file > file

What do you expect to see in the test file?

The result is that the file is completely empty.

c – Output redirection gotchas cat-file-empty

Why?

By default, when parsing your command, the kernel will not execute the commands sequentially.

It means that it won’t wait for the end of the sed command to open your new file and to write the content to it.

Instead, the kernel is going to open your file, erase all the content inside it, and wait for the result of your sed operation to be processed.

As the sed operation is seeing an empty file (because all the content was erased by the output redirection operation), the content is empty.

As a consequence, nothing is appended to the file, and the content is completely empty.

In order to redirect the output to the same file, you may want to use pipes or more advanced commands such as

command … input_file > temp_file  &&  mv temp_file input_file

Protecting a file from being overwritten

In Linux, it is possible to protect files from being overwritten by the “>” operator.

You can protect your files by setting the “noclobber” parameter on the current shell environment.

$ set -o noclobber

It is also possible to restrict output redirection by running

$ set -C

Note: to re-enable output redirection, simply run set +C

noclobber

As you can see, the file cannot be overridden when setting this parameter.

If I really want to force the override, I can use the “>|” operator to force it.

override-1

What is Input Redirection on Linux?

Input redirection is the act of redirecting the input of a process to a given device (or virtual device) so that it starts reading from this device and not from the default one assigned by the Kernel.

a – How does input redirection works?

As an instance, when you are opening a terminal, you are interacting with it with your keyboard.

However, there are some cases where you might want to work with the content of a file because you want to programmatically send the content of the file to your command.

input-redirection-diagram

To redirect the standard input on Linux, you have to use the “<” operator.

As an example, let’s say that you want to use the content of a file and run a special command on them.

In this case, I am going to use a file containing domains, and the command will be a simple sort command.

In this way, domains will be sorted alphabetically.

With input redirection, I can run the following command

cat-input-redirect

If I want to sort those domains, I can redirect the content of the domains file to the standard input of the sort function.

$ sort < domains

sort-example

With this syntax, the content of the domains file is redirected to the input of the sort function. It is quite different from the following syntax

$ sort domains

Even if the output may be the same, in this case, the sort function takes a file as a parameter.

In the input redirection example, the sort function is called with no parameter.

As a consequence, when no file parameters are provided to the function, the function reads it from the standard input by default.

In this case, it is reading the content of the file provided.

b – Redirecting standard input with a file containing multiple lines

If your file is containing multiple lines, you can still redirect the standard input from your command for every single line of your file.

b – Redirecting standard input with a file containing multiple lines multiline1

Let’s say for example that you want to have a ping request for every single entry in the domains file.

By default, the ping command expects a single IP or URL to be pinged.

You can, however, redirect the content of your domain’s file to a custom function that will execute a ping function for every entry.

$ ( while read ip; do ping -c 2 $ip; done ) < ips

b – Redirecting standard input with a file containing multiple lines

c – Combining input redirection with output redirection

Now that you know that standard input can be redirected to a command, it is useful to mention that input and output redirection can be done within the same command.

Now that you are performing ping commands, you are getting the ping statistics for every single website on the domains list.

The results are printed on the standard output, which is in this case the terminal.

But what if you wanted to save the results to a file?

This can be achieved by combining input and output redirections on the same command.

$ ( while read ip; do ping -c 2 $ip; done ) < domains > stats.txt

Great! The results were correctly saved to a file and can be analyzed later on by other teams in your company.

d – Discarding standard output completely

In some cases, it might be handy to discard the standard output completely.

It may be because you are not interested in the standard output of a process or because this process is printing too many lines on the standard output.

To discard standard output completely on Linux, redirect the standard output to /dev/null.

Redirecting to /dev/null causes data to be completely discarded and erased.

$ cat file > /dev/null

Note: Redirecting to /dev/null does not erase the content of the file but it only discards the content of the standard output.

What is standard error redirection on Linux?

Finally, after input and output redirection, let’s see how standard error can be redirected.

a – How does standard error redirection work?

Very similarly to what we saw before, error redirection is redirecting errors returned by processes to a defined device on your host.

For example, if I am running a command with bad parameters, what I am seeing on my screen is an error message and it has been processed via the file descriptor responsible for error messages (fd[2]).

Note that there are no trivial ways to differentiate an error message from a standard output message in the terminal, you will have to rely on the programmer sending error messages to the correct file descriptor.

error-redirection-diagram

To redirect error output on Linux, use the “2>” operator

$ command 2> file

Let’s use the example of the ping command in order to generate an error message on the terminal.

Now let’s see a version where the error output is redirected to an error file.

As you can see, I used the “2>” operator to redirect errors to the “error-file” file.

If I were to redirect only the standard output to the file, nothing would be printed to it.

As you can see, the error message was printed to my terminal and nothing was added to my “normal-file” output.

b – Combining standard error with standard output

In some cases, you may want to combine the error messages with the standard output and redirect it to a file.

It can be particularly handy because some programs are not only returning standard messages or error messages but a mix of two.

Let’s take the example of the find command.

If I am running a find command on the root directory without sudo rights, I might be unauthorized to access some directories, like processes that I don’t own for example.

permission-denied

As a consequence, there will be a mix of standard messages (the files owned by my user) and error messages (when trying to access a directory that I don’t own).

In this case, I want to have both outputs stored in a file.

To redirect the standard output as well as the error output to a file, use the “2<&1” syntax with a preceding “>”.

$ find / -user junosnotes > file 2>&1

Alternatively, you can use the “&>” syntax as a shorter way to redirect both the output and the errors.

$ find / -user junosnotes &> file

So what happened here?

When bash sees multiple redirections, it processes them from left to right.

As a consequence, the output of the find function is first redirected to the file.

Next, the second redirection is processed and redirects the standard error to the standard output (which was previously assigned to the file).

multiple-redirections

multiple-redirections 2

What are pipelines on Linux?

Pipelines are a bit different from redirections.

When doing standard input or output redirection, you were essentially overwriting the default input or output to a custom file.

With pipelines, you are not overwriting inputs or outputs, but you are connecting them together.

Pipelines are used on Linux systems to connect processes together, linking standard outputs from one program to the standard input of another.

Multiple processes can be linked together with pipelines (or pipes)

pipelines-linux-2

Pipes are heavily used by system administrators in order to create complex queries by combining simple queries together.

One of the most popular examples is probably counting the number of lines in a text file, after applying some custom filters on the content of the file.

Let’s go back the domains file we created in the previous sections and let’s change their country extensions to include .net domains.

Now let’s say that you want to count the numbers of .com domains in the file.

How would you perform that? By using pipes.

First, you want to filter the results to isolate only the .com domains in the file. Then, you want to pipe the result to the “wc” command in order to count them.

Here is how you would count .com domains in the file.

$ grep .com domains | wc -l

Here is what happened with a diagram in case you still can’t understand it.

counting-domains

Awesome!

Conclusion

In today’s tutorial, you learned what input and output redirection is and how it can be effectively used to perform administrative operations on your Linux system.

You also learned about pipelines (or pipes) that are used to chain commands in order to execute longer and more complex commands on your host.

If you are curious about Linux administration, we have a whole category dedicated to it on JunosNiotes, so make sure to check it out!

How To Install AutoFS on Linux

Whether you are an experienced system administrator or just a regular user, you have probably already mounted drives on Linux.

Drives can be local to your machine or they can be accessed over the network by using the NFS protocol for example.

If you chose to mount drives permanently, you have probably added them to your fstab file.

Luckily for you, there is a better and more cost effective way of mounting drives : by using the AutoFS utility.

AutoFS is a utility that mount local or remote drives only when they are accessed : if you don’t use them, they will be unmounted automatically.

In this tutorial, you will learn how you can install and configure AutoFS on Linux systems.

Prerequisites

Before starting, it is important for you to have sudo privileges on your host.

To verify it, simply run the “sudo” command with the “-v” option : if you don’t see any options, you are good to go.

$ sudo -v

If you don’t have sudo privileges, you can follow this tutorial for Debian based hosts or this tutorial for CentOS based systems.

Installing AutoFS on Linux

Before installing the AutoFS utility, you need to make sure that your packages are up-to-date with repositories.

$ sudo apt-get update

Now that your system is updated, you can install AutoFS by running the “apt-get install” command with the “autofs” argument.

$ sudo apt-get install autofs

When installing the AutoFS package, the installation process will :

  • Create multiple configuration files in the /etc directory such as : auto.master, auto.net, auto.misc and so on;
  • Will create the AutoFS service in systemd;
  • Add the “automount” entry to your “nsswitch.conf” file and link it to the “files” source

Right after the installation, make sure that the AutoFS service is running with the “systemctl status” command

$ sudo systemctl status autofs

Installing AutoFS on Linux autofs-service

You can also enable the AutoFS service for it to be run at startup

$ sudo systemctl enable autofs

Now that AutoFS is correctly installed on your system, let’s see how you can start creating your first map.

How AutoFS works on Linux

Maps” are a key concept when it comes to AutoFS.

In AutoFS, you are mapping mount points with files (which is called an indirect map) or a mount point with a location or a device.

In its default configuration, AutoFS will start by reading maps defined in the autofs.master file in the /etc directory.

From there, it will start a thread for all the mount points defined in the map files defined in the master file.

How AutoFS works on Linux autofs

Starting a thread does not mean that the mount point is mounted when you first start AutoFS : it will only be mounted when it is accessed.

By default, after five minutes of inactivity, AutoFS will dismount (or unmount) mount points that are not used anymore.

Note : configuration parameters for AutoFS are available in the /etc/autofs.conf

Creating your first auto map file

Now that you have an idea on how AutoFS works, it is time for you to start creating your very first AutoFS map.

In the /etc directory, create a new map file named “auto.example“.

$ sudo touch /etc/auto.example

The goal of this map file will be to mount a NFS share located on one computer on the network.

The NFS share is located at the IP 192.168.178.29/24 on the local network and it exports one drive located at /var/share.

Before trying to automount the NFS share, it is a good practice to try mounting it manually as well as verifying that you can contact the remote server.

$ ping 192.168.178.29

Creating a direct map

The easiest mapping you can create using AutoFS is called a direct map or a direct mapping.

A direct map directly associates one mount point with a location (for example a NFS location)

Creating your first auto map file direct-mapping

As an example, let’s say that you want to mount a NFS share at boot time on the /tmp directory.

To create a direct map, edit your “auto.example” file and append the following content in it :

# Creating a direct map with AutoFS

# <mountpoint>    <options>    <remote_ip>:<location>   

/tmp              -fstype=nfs  192.168.178.29:/var/share

Now, you will need to add the direct map to your “auto.master” file.

To specify that you are referencing a direct map, you need to use the “-” notation

# Content of the auto.master file

/-    auto.example

direct-map

Now that your master file is modified, you can restart the AutoFS service for the changes to be effective.

$ sudo systemctl restart autofs

$ cd /tmp

Congratulations, you should now be able to access your files over NFS via direct mapping.

Creating a direct map tmp-nfs

Creating an indirect mapping

Now that you have discovered direct mappings, let’s see how you can use indirect mappings in order to mount remote location on your filesystem.

Indirect mappings use the same syntax as direct mappings with one small difference : instead of mounting locations directly to the mountpoint, you are mounting it in a location in this mountpoint.

Creating an indirect mapping

To understand it, create a file named “auto.nfs” and paste the following content in it

nfs    -fstype=nfs  192.168.178.29:/var/share

As you can see, the first column changed : in a direct map, you are using the path to the mountpoint (for example /tmp), but with an indirect map you are specifying the key.

The key will represent the directory name located in the mount point directory.

Edit your “auto.master” file and add the following content in it

/tmp   /etc/auto.nfs

Creating an indirect mapping autonfs

Restart your AutoFS service and head over to the “tmp” directory

$ sudo systemctl restart autofs

$ cd /tmp

By default, there won’t be anything displayed if you list the content of this directory : remember, AutoFS will only mount the directories when they are accessed.

In order for AutoFS to mount the directory, navigate to the directory named after the key that you specified in the “auto.nfs” file (called “nfs” in this case)

$ cd nfs

Awesome!

Your mountpoint is now active and you can start browsing your directory.

Mapping distant home directories

Now that you understand a bit more about direct and indirect mappings, you might ask yourself one question : what’s the point of having indirect mapping when you can simply map locations directly?

In order to be useful, indirect maps are meant to be used with wildcard characters.

One major use-case of the AutoFS utility is to be able to mount home directories remotely.

However, as usernames change from one user to another, you won’t be able to have a clean and nice-looking map file, you would have to map every user in a very redundant way.

# Without wildcards, you have very redundant map files

/home/antoine  <ip>:/home/antoine
/home/schkn    <ip>:/home/schkn
/home/devconnected <ip>:/home/devconnected

Luckily for you, there is a syntax that lets your dynamically create directories depending on what’s available on the server.

To illustrate this, create a new file named “auto.home” in your /etc directory and start editing it.

# Content of auto.home

*    <ip>:/home/&

In this case, there are two wilcards and it simply means that all the directories found in the /home directory on the server will be mapped to a directory of the same name on the client.

To illustrate this, let’s pretend that we have a NFS server running on the 192.168.178.29 IP address and that it contains all the home directories for our users.

# Content of auto.home

*   192.168.178.29:/home/&

Save your file and start editing your auto.master file in order to create your indirect mapping

$ sudo nano /etc/auto.master

# Content of auto.master

/home     /etc/auto.home

Save your master file and restart your AutoFS service for the changes to be applied.

$ sudo systemctl restart autofs

Now, you can head over to the /home directory and you should be able to see the directories correctly mounted for the users.

Note : if you see nothing in the directory, remember that you may need to access the directory one time for it to be mounted by AutoFS

Mapping distant home directories home-dir

Mapping and discovering hosts on your network

If you paid attention to the auto.master file, you probably noticed that there is an entry for the /net directory with a value “-hosts“.

The “-hosts” parameter is meant to represent all the entries defined in the /etc/hosts file.

As a reminder, the “hosts” file can be seen as a simple and local DNS resolver that associates a set of IPs with hostnames.

As an example, let’s define an entry for the NFS server into the /etc/hosts file by filling the IP and the hostname of the machine.

Mapping and discovering hosts on your network dns-resolver

First of all, make sure that some directories are exported on the server by running the “showmount” command on the client.

$ sudo showmount -e <server>

Mapping and discovering hosts on your network showmount

Now that you made sure that some directories are exported, head over to your “auto.master” file in /etc and add the following line.

# Content of auto.master

/net   -hosts

Save your file and restart your AutoFS service for the changes to be applied.

$ sudo systemctl restart autofs

That’s it!

Now your NFS share should be accessible in the /net directory under a directory named after your server hostname.

$ cd /net/<server_name>

$ cd /net/<server_ip>
Note : remember that you will need to directly navigate in the directory for it to be mounted. You won’t see it by simply listing the /net directory on the first mount.

Troubleshooting

In some cases, you may have some troubles while setting up AutoFS : when a device is busy or when you are not able to contact a remote host for example.

  • mount/umount : target is busy

As Linux is a multi-user system, you might have some users browsing some locations that you are trying to mount or unmount (using AutoFS or not)

If you want to know who is navigating the folder or who is using a file, you have to use the “lsof” command.

$ lsof +D <directory>
$ lsof <file>

Troubleshooting lsof

Note : the “+D” option is used in order to list who is using the resource recursively.
  • showmount is hanging when configuring host discovery

If you tried configuring host discovery by using the “-hosts” parameter, you might have verified that your remote hosts are accessible using the “showmount” command.

However, in some cases, the “showmount” command simply hangs as it is unable to contact the remote server.

Most of the time, the server firewall is blocking the requests made by the client.

If you have access to the server, you try to inspect the logs in order to see if the firewall (UFW for example) is blocking the requests or not.

firewall-blocking

  • Debugging using the automount utility

On recent distributions, the autofs utility is installed as a systemd service.

As a consequence, you can inspect the autofs logs by using the “journalctl” command.

$ sudo journalctl -u autofs.service

You can also use the “automount” utility in order to debug the auto mounts done by the service.

$ sudo systemctl stop autofs

$ sudo automount -f -v

Conclusion

In this tutorial, you learnt about the AutoFS utility : how it works and the differences between direct and indirect maps.

You also learnt that it can be configured in order to setup host discovery : out of the box, you can connect to all the NFS shares of your local network which is a very powerful tool.

Finally, you have seen how you can create indirect maps in order to automatically create home directories on the fly.

If you are interested in Linux system administration, we have a complete section dedicated to it, so make sure to have a look!

Arping Command on Linux Explained

As a network administrator, you are probably already very familiar with the ARP protocol.

ARP is commonly used by layer two devices in order to discover as well as communicating with each other easily.

When you are dealing with a small office network, you might be tempted to ping hosts in order to verify that they are available.

If you are using the ICMP protocol, you might be aware that you are actually performing ARP requests in order to probe devices on your network.

If you are looking for a more straightforward way to create ARP pings, you might be interested in the arping command.

In this tutorial, we are going to focus on the arping command : how to install it and how to use it effectively.

Prerequisites

In order to install the arping command on your system, you will obviously need sudo privileges on your server.

In order to check if you are sudo or not, you can simply execute the following command

$ groups

user sudo

If this is not the case, you can read our guide on getting sudo privileges for Debian or CentOS hosts.

In order to install the arping command on your server, execute the “apt-get install” command and specify the “arping” package.

$ sudo apt-get install arping

Installing arping on Linux arping

Now that the command is installed, you can execute the “arping” command in order to check the current version used.

$ arping -v

ARPing 2.19, by Thomas Habets <thomas@habets.se>

Great!

The arping command is now installed on your server.

By default, the arping command is going to send an ARP (or ICMP) request every second, but it can obviously be configured.

Using arping to discover hosts

First of all, as any device communicating over Ethernet, your device has an internal ARP table used to communicate over the network.

In order to see your current ARP entries, you can simply execute the “arp” command with the “-a” option for all devices.

$ arp -a

When using the ARP command, you are presented with a list of hostnames, followed by IPs and MAC addresses.

Using arping to discover hosts arp-table

In this case, I am presented with the only entry in my ARP table : a router accessible via the 192.168.178.1 IP address.

However, I might be interested in finding other hosts on my local network : to achieve that, you are going to use the arping command.

Pinging hosts using IP addresses

In order to ping hosts over your network, you can simply use the “arping” command and specify the IP address to be pinged.

Additionally, you can specify the number of pings to be sent using the “-c” option for “count”.

$ arping -c 2 <ip_address>
Note : if you are not sure about the way of finding your IP address on Linux, we have a complete guide on the subject.

For example, using the “192.168.178.27” IP address over your local network, you would execute the following command

Pinging hosts using IP addresses arping-example

As you can see, if you are getting response pings, you are presented with the MAC address of the corresponding device.

Note that using the arping command will not automatically update your ARP table : you would have to use a command such as ping in order to update it.

$ arp -a

Pinging hosts using IP addresses arp-update

Awesome, you successfully used the arping command in order to issue ARP requests over the network!

ARP timeouts using arping

If the arping command is not able to resolve the IP address of the target defined, you will get an ARP timeout.

As an example, executing an ARP request on an unknown host would give you the following output

$ arping -c 5 <ip_address>

ARP timeouts using arping-timeout

As you can see, in some cases, you will be presented with a warning if you don’t specify any network interface.

This is quite normal because the arping command expects a network interface to be specified.

If you were to deal with a router, or if you chose to install your Linux server as a router, two network interface cards can be installed in order to route to two different networks.

If this is the case, the arping needs to know the network interface it needs to use in order to send the ARP ping.

As you can see, the arping command will try to “guess” the network interface if it is not provided with one.

Specifying the network interface

If you have multiple network interfaces on your server, the arping won’t be able to “guess” the network interface card to be used.

As a consequence, you might get an error message stating that the arping was not able to guess the correct one.

Specifying the network interface suitable-device-guess

In order to specify the network interface to be used, you will have to use the “-I” option followed by the name of the network interface.

If you need some help on how to enumerate network interfaces, you can use this guide on finding your IP address on Linux.

$ arping -I <interface_name> <ip_address>

If our interface is named “enp0s3”, the command would be the following one :

$ arping -I enp0s3 192.168.178.22

Specifying the network interface arping-network-interface

Awesome, you have pinged your distant server and you have specified the network interface to be used!

Sending ARP pings from Source MAC

In some cases, you may want to specify the source MAC address you are sending packets from.

In order to achieve that, you need to execute the “arping” command with the “-s” option for “source” followed by the MAC address you want to ping.

$ arping -c 2 -s 00:60:70:12:34:56 <ip_address>

In this case, you have two possibilities :

  • You are the owner of the MAC address and you can simply use the “-s” option.
  • You are not the owner of the MAC address and you are trying to spoof the MAC address. In this case, you need to use the promiscuous mode. As a short reminder, the promiscuous mode is set to transmit all frames received by the NIC rather than the ones it was meant to receive.

In order to enable the promiscuous mode with the “arping” command, you need to use the “-p” option.

Using the options we used previously, this would lead us to the following command.

$ arping -c 2 -s 00:60:70:12:34:56 -p <ip_address>

Conclusion

In this tutorial, you learnt how you can easily use the arping in order to ping IP addresses on your local network.

Using arping, you are able to populate your local ARP cache with the matching MAC address.

You also learnt that you are able to “spoof” your MAC address by using the promiscuous mode.

If you are interested in Linux System Administration, we have a complete section dedicated to it on the website, so make sure to check it out!

How To Install Samba on Debian 10 Buster

If you are working on a small to medium entreprise network, you probably have dozens of drives and printers that need to be shared.

Besides the NFS protocol, there are plenty of other network protocols that can be used in order to share resources over a network.

The CIFS, short for Common Internet File System, is a network filesystem protocol used to share resources among multiple hosts, sharing the same operating system or not.

The CIFS, also known as the SMB protocol, is implemented by one popular tool : the Samba server.

Started in 1991, Samba was developed in the early days in order to ease the interoperability of Unix and Windows based systems.

In this tutorial, we are going to focus on the Samba installation and configuration for your network.

Prerequisites

In order to install new packages on your system, you will need to be a user with elevated permissions.

To check if you are already a sudo user, you can run the “groups” command and check if “sudo” belongs to the list.

$ groups

user sudo netdev cdrom

If you don’t belong to the sudo group, you can check one of our tutorials in order to gain sudo privileges for Debian instances.

Now that you have sudo privileges, let’s jump right into the Samba server installation.

Installing Samba on Debian

Before installing Samba, you will need to make sure that your packages are up-to-date with the Debian repositories.

$ sudo apt-get update

Now that your system is up-to-date, you can run the “apt-get install” command on the “samba” package.

$ sudo apt-get install samba

When installing Samba, you will be presented with the following screen.

Installing Samba on Debian samba

In short, this window is used in order to configure retrieval of NetBIOS name servers over your network.

Nowadays, your entreprise network is most likely using DNS name servers in order to store static information about hostnames over your network.

As a consequence, you are most likely not using a WINS server, so you can select the “No” option.

When resuming the installation, APT will unpack and install the packages needed for Samba.

Additionnally, a “sambashare” group will be created.

After the installation, you can check the version used on your system by running the “samba” command with the “-v” option.

$ samba -V

Installing Samba on Debian samba-version

You can also verify that the Samba server is running by checking the status of the Samba SMB Daemon with systemctl.

$ systemctl status smbd

Installing Samba on Debian samba-service

Great, Samba is now correctly installed on your Debian server!

Opening Samba Ports on your firewall

This section only applies if you are using UFW or FirewallD on your server.

In order for Samba to be reachable from Windows and Linux hosts, you have to make sure that ports 139 and 445 are open on your firewall.

On Debian and Ubuntu, you are probably using the UFW firewall.

In order to open ports on your UFW firewall, you have to use the “allow” command on ports 139 and 445.

$ sudo ufw allow 139
$ sudo ufw allow 445

$ sufo ufw status

Opening Samba Ports on your firewall ufw-status

If you are working on a CentOS or a RHEL server, you will have to use the “firewall-cmd” in order to open ports on your computer.

$ sudo firewall-cmd --permanent --add-port=139/tcp
success
$ sudo firewall-cmd --permanent --add-port=445/tcp
success
$ sudo firewall-cmd --reload
success

Opening Samba Ports on your firewall-centos

Configuring Samba on Debian

Now that your Samba is correctly installed, it is time to configure it in order to configure it in order to be able to export some shares.

Note : Samba can also be configured in order to act as a domain controller (like Active Directory) but this will be explained in another tutorial.

By default, the Samba configuration files are available in the “/etc/samba” folder.

Configuring Samba on Debian conf-folder

By default, the Samba folder contains the following entries :

  • gdbcommands : a file containing a set of entries for the GDB debugger (won’t be used at all here);
  • smb.conf : the main Samba configuration file;
  • tls : a directory used in order to store TLS and SSL information about your Samba server.

For this section, we are going to focus on the content of the smb.conf file.

The Samba configuration file is composed of different sections :

  • global : as its name indicates, it is used in order to define Samba global parameters such as the workgroup (if you are using Windows), the log location, as well as PAM password synchronization if any;
  • shares definitions : in this section, you will list the different shares exported by the Samba server.

Defining the Windows workgroup

If you plan on including the Samba server into a Windows workgroup, you will need to determine the workgroup your computers belong to.

If you are working on a Unix-only network, you can skip this section and jump right into share definition.

Note : if you are using a domain controller, those settings do not apply to you.

In order to find your current workgroup, head over to the Windows Start Menu, and search for “Show which workgroup this computer is on”.

Defining the Windows workgroup

Select the option provided by the search utility and you should be able to find your workgroup in the next window.

Defining the Windows workgroup-2

In this case, the workgroup name is simply “WORKGROUP“.

However, you will have to make sure that this name is reflected in the Samba configuration file.

Defining the Windows workgroup-3

Now that your workgroup is properly configured, let’s start by defining simple share definitions for your Samba server.

Defining Samba share definitions

On Samba, a share is defined by specifying the following fields :

  • Share name : the name of the share as well as the future address for your share (the share name is to be specified into brackets);
  • Share properties : the path to your share, if it is public, if it can be browsed, if you can read files or create files and so on.

In order to start simply, let’s create a Samba share that will be publicly available to all machines without authentication.

Note : it is recommended to setup Samba authentication if you are exporting shares containing sensitive or personal information.

Creating a public Samba share

First of all, you will need to decide on the folder to be exported on your system, for this tutorial we are going to choose “/example”.

In order for users to be able to write files to the share, they will need to have permissions on the share.

However, we are not going to set full permissions to all users on the folder, we are going to create a system account (that has write permissions) and we are going to force user to use this account when logging to Samba.

In order to create a system account, use the “useradd” command with the “-r” option for system accounts.

$ sudo useradd -rs /bin/false samba-public

$ sudo chown samba-public /example

$ sudo chmod u+rwx /example

In order to create a public Samba share, head over to the bottom of your Samba configuration file and add the following section.

$ nano /etc/samba/smb.conf

[public]
   path = /example
   available = yes
   browsable = yes
   public = yes
   writable = yes
   force user = samba-public

Here is an explanation of all the properties specified in this Samba share definition :

  • path : pretty self-explanatory, the path on your filesystem to be exported with Samba;
  • available : meaning that the share will be exported (you can choose to have shares defined but not exported);
  • browsable : meaning that the share will be public in network views (such as the Windows Network view for example);
  • public : synonym for “guest ok”, this parameter means that everyone can export this share;
  • writable : meaning that all users are able to create files on the share.
  • force user : when logging, users will take the identify of the “samba-public” account.

Before restarting your smbd service, you can use the “testparm” in order to check that your configuration is syntactically correct.

$ testparm

Creating a public Samba share testparm

As you can see, no syntax errors were raised during the configuration verification, so we should be good to go.

Now that your share definition is created, you can restart your smbd service in order for the changes to be applied.

$ sudo systemctl restart smbd

$ sudo systemctl status smbd

Your share should now be accessible : in order to verify it, you can install the “samba-client” package and list the shares exported on your local machine.

$ sudo apt-get install smbclient

$ smbclient -L localhost
Note : you will be asked to provide a password for your workgroup. In most cases, you have no password for your workgroup, you can simply press Enter.

Creating a public Samba share smbclient

Connecting to Samba from Linux

In order to be able to mount CIFS filesystems, you have to install CIFS utilities on your system.

$ sudo apt-get install cifs-utils

Now that CIFS utilities are installed, you will be able to mount your filesystem using the mount command.

$ sudo mount -t cifs //<server_ip>/<share_name> <mountpoint>

Using our previous example, our exported share was named “public” and it was available on the 192.168.178.35 IP address.

Note : you can follow this tutorial if you are not sure how you can find your IP address on Linux.

If we were to mount the filesystem on the “/mnt” mountpoint, this would give

$ sudo mount -t cifs //192.168.178.35/public /mnt -o uid=devconnected

Password for root@//192.168.178.35/public : <no_password>

Now that your drive is mounted, you can access it like any other filesystem and start creating files on it.

Congratulations, you successfully mounted a CIFS drive on Linux!

Connecting to Samba from Windows

If you are using a Windows host, it will be even easier for you to connect to a Samba share.

In the Windows Search menu, look for the “Run” application.

Connecting to Samba from Windows run-app

In the Run windows, connect to the Samba share using the same set of information than the Linux setup.

Be careful : on Windows, you have to use backslashes instead of slashes.

Connecting to Samba from Windows run-app-2

When you are done, simply click on “Ok” in order to navigate your share!

Awesome, you successfully browsed your Samba on Windows!

Securing Samba shares

In the previous sections, we have created a public share.

However, in most cases, you may want to build secure share that are accessible only by a restricted number of users on your network.

By default, Samba authentication is separated from Unix authentication : this statement means that you will have to create separate Samba credentials for your users.

Note : you may choose to have Samba built as an AD/DC but this would be a completely different tutorial.

In order to create a new Samba, you need to use the “smbpasswd” command and specify the name of the user to be created.

$ smbpasswd <user>
Note : the user you are trying to create with Samba needs to have a Unix account already configured on the system.

Now that your user is created, you can edit your Samba configuration file in order to make your share secure.

$ nano /etc/samba/smb.conf

[private]
   path = /private
   available = yes
   browsable = yes
   public = no
   writable = yes
   valid users = <user>

Most of the options were already described in the previous section, except for the “valid users” one which, as its name specifies, authorizes the Samba access to a restricted list of users.

Again, you can test your Samba configuration with the “testparm” command and restart your Samba service if everything is okay.

$ testparm

$ sudo systemctl restart smbd

$ sudo systemctl status smbd

Now that your drive is secured, it is time for you to start accessing it from your remote operating systems.

Connecting to secure Samba using Windows

On Windows, you will have to use the same procedure than the previous step : execute the “Run” application and type the address of your share drive.

Connecting to secure Samba using Windows private

When clicking on “Ok”, you will be presented with a box asking for your credentials : you have to use the credentials you defined in the previous section with smbpasswd.

Connecting to secure Samba using windows-pass-1

If you provided the correct password, you should be redirected to your network drive, congratulations!

Connecting to secure Samba using Linux

In order to connect to a secure Samba share using Linux, you have to use the “mount” command and provide the address of the share as well as the mount point to be used.

$ sudo mount -t cifs //<share_ip>/<share_name> <mount_point> -o username=<user>

Using the example of our “private” share on the 192.168.178.35 IP address, this would result in the following command :

$ sudo mount -t cifs //192.168.178.35/private /mnt -o username=user

Password for user@//192.168.178.35/private: <provide_password>

That’s it!

Your drive should now be correctly mounted.

You can verify that it was correctly mounted with the “findmnt” command that lists mounted filesystems.

$ findmnt /mnt

Connecting to secure Samba using Linux findmnt

Congratulations, you successfully mounted a secure Samba share on your server!

Conclusion

In this tutorial, you learnt how you can easily install and configure a Samba server in order to share your drives.

You also learnt that you can tweak Samba share options in order to make your shares secure, whether you are using Windows or Linux.

Samba is an important tool working on the interoperability of operating systems : if you are interested in the Samba project, you should definitely check their website.

They are also providing a free alternative to Active Directory where Samba can be configured to act as a domain controller.

If you are interested in Linux System Administration, we have a complete section dedicated to it on the website, so make sure to check it out!

Linux Logging Complete Guide

As a Linux system administrator, inspecting log files is one of the most common tasks that you may have to perform.

Linux logs are crucial : they store important information about some errors that may happen on your system.

They might also store information about who’s trying to access your system, what a specific service is doing, or about a system crash that happened earlier.

As a consequence, knowing how to locatemanipulate and parse log files is definitely a skill that you have to master.

In this tutorial, we are going to unveil everything that there is to know about Linux logging.

You will be presented with the way logging is architectured on Linux systems and how different virtual devices and processes interact together to log entries.

We are going to dig deeper into the Syslog protocol and how it transitioned from syslogd (on old systems) to journalctl powered by systemd on recent systems.

Linux Logging Types

When dealing with Linux logging, there are a few basics that you need to understand before typing any commands in the terminal.

On Linux, you have two types of logging mechanisms :

  • Kernel logging: related to errors, warning or information entries that your kernel may write;
  • User logging: linked to the user space, those log entries are related to processes or services that may run on the host machine.

By splitting logging into two categories, we are essentially unveiling that memory itself is divided into two categories on Linux : user space and kernel space.

Linux Logging Types linux-spaces

Kernel Logging

Let’s start first with logging associated with the kernel space also known as the Kernel logging.

On the kernel space, logging is done via the Kernel Ring Buffer.

The kernel ring buffer is a circular buffer that is the first datastructure storing log messages when the system boots up.

When you are starting your Linux machine, if log messages are displayed on the screen, those messages are stored in the kernel ring buffer.

Kernel Logging

Kernel logs during boot process

The Kernel logging is started before user logging (managed by the syslog daemon or by rsyslog on recent distributions).

The kernel ring buffer, pretty much like any other log files on your system can be inspected.

In order to open Kernel-related logs on your system, you have to use the “dmesg” command.

Note : you need to execute this command as root or to have privileged rights in order to inspect the kernel ring buffer.
$ dmesg

Kernel Logging dmesg

As you can see, from the system boot until the time when you executed the command, the kernel keeps track of all the actions, warnings or errors that may happen in the kernel space.

If your system has trouble detecting or mounting a disk, this is probably where you want to inspect the errors.

As you can see, the dmesg command is a pretty nice interface in order to see kernel logs, but how is the dmesg command printing those results back to you?

In order to unveil the different mechanisms used, let’s see which processes and devices take care of Kernel logging.

Kernel Logging internals

As you probably heard it before, on Linux, everything is a file.

If everything is a file, it also means that devices are files.

On Linux, the kernel ring buffer is materialized by a character device file in the /dev directory and it is named kmsg.

$ ls -l /dev/ | grep kmsg

Kernel Logging internals kmsg

If we were to depict the relationship between the kmsg device and the kernel ring buffer, this is how we would represent it.

kernel-logging-internals

As you can see, the kmsg device is an abstraction used in order to read and write to the kernel ring buffer.

You can essentially see it as an entrypoint for user space processes in order to write to the kernel ring buffer.

However, the diagram shown above is incomplete as one special file is used by the kernel in order to dump the kernel log information to a file.

Kernel Logging internals

If we were to summarize it, we would essentially state that the kmsg virtual device acts as an entrypoint for the kernel ring buffer while the output of this process (the log lines) are printed to the /proc/kmsg file.

This file can be parsed by only one single process which is most of the time the logging utility used on the user space. On some distributions, it can be syslogd, but on more recent distributions it is integrated with rsyslog.

The rsyslog utility has a set of embedded modules that will redirect kernel logs to dedicated files on the file system.

Historically, kernel logs were retrieved by the klogd daemon on previous systems but it has been replaced by rsyslog on most distributions.

Kernel Logging internals klogd

klogd utility running on Debian 4.0 Etch

On one hand, you have logging utilities reading from the ring buffer but you also have user space programs writing to the ring buffer : systemd (with the famous systemd-journal) on recent distributions for example.

Now that you know more about Kernel logging, let’s see how logging is done on the user space.

User side logging with Syslog

Logging on the userspace is quite different from logging on the kernel space.

On the user side, logging is based on the Syslog protocol.

Syslog is used as a standard to produce, forward and collect logs produced on a Linux instance.

Syslog defines severity levels as well as facility levels helping users having a greater understanding of logs produced on their computers.

Logs can later on be analyzed and visualized on servers referred as Syslog servers.

User side logging with Syslog-card

In short, the Syslog protocol is a protocol used to define the log messages are formatted, sent and received on Unix systems.

Syslog is known for defining the syslog format that defines the format that needs to be used by applications in order to send logs.

This format is well-known for defining two important terms : facilities and priorities.

Syslog Facilities Explained

In short, a facility level is used to determine the program or part of the system that produced the logs.

On your Linux system, many different utilities and programs are sending logs. In order to determine which process sent the log in the first place, Syslog defines numbers, facility numbers, that are used by programs to send Syslog logs.

There are more than 23 different Syslog facilities that are described in the table below.

Numerical Code Keyword Facility name
0 kern Kernel messages
1 user User-level messages
2 mail Mail system
3 daemon System Daemons
4 auth Security messages
5 syslog Syslogd messages
6 lpr Line printer subsystem
7 news Network news subsystem
8 uucp UUCP subsystem
9 cron Clock daemon
10 authpriv Security messages
11 ftp FTP daemon
12 ntp NTP subsystem
13 security Security log audit
14 console Console log alerts
15 solaris-cron Scheduling logs
16-23 local0 to local7 Locally used facilities

Most of those facilities are reserved to system processes (such as the mail server if you have one or the cron utility). Some of them (from the facility number 16 to 23) can be used by custom Syslog client or user programs to send logs.

Syslog Priorities Explained

Syslog severity levels are used to how severe a log event is and they range from debug, informational messages to emergency levels.

Similarly to Syslog facility levels, severity levels are divided into numerical categories ranging from 0 to 7, 0 being the most critical emergency level.

Again, here is a table for all the priority levels available with Syslog.

Here are the syslog severity levels described in a table:

Value Severity Keyword
0 Emergency emerg
1 Alert alert
2 Critical crit
3 Error err
4 Warning warning
5 Notice notice
6 Informational info
7 Debug debug

Syslog Architecture

Syslog also defines a couple of technical terms that are used in order to build the architecture of logging systems :

  • Originator : also known as a “Syslog client”, an originator is responsible for sending the Syslog formatted message over the network or to the correct application;
  • Relay : a relay is used in order to forward messages over the network. A relay can transform the messages in order to enrich it for example (famous examples include Logstash or fluentd);
  • Collector : also known as “Syslog servers”, collectors are used in order to store, visualize and retrieve logs from multiple applications. The collector can write logs to a wide variety of different outputs : local files, databases or caches.

Syslog Architecture syslog

As you can see, the Syslog protocol follows the client-server architecture we have seen in previous tutorials.

One Syslog client creates messages and sends it to optional local or distant relays that can be further transferred to Syslog servers.

Now that you know how the Syslog protocol is architectured, what about our own Linux system?

Is it following this architecture?

Linux Local Logging Architecture

Logging on a local Linux system follows the exact principles we have described before.

Without further ado, here is the way logging is architectured on a Linux system (on recent distributions)

Linux Local Logging Architecture linux-logging-2

Following the originator-relay-collector architecture described before, in the case of a local Linux system :

  • Originators are client applications that may embed syslog or journald libraries in order to send logs;
  • No relays are implemented by default locally;
  • Collectors are rsyslog and the journald daemon listening on predefined sockets for incoming logs.

So where are logs stored after being received by the collectors?

Linux Log File Location

On your Linux system, logs are stored in the /var/log directory.

Logs in the /var/log directory are split into the Syslog facilities that we saw earlier followed by the log suffix : auth.log, daemon.log, kern.log or dpkg.log.

If you inspected the auth.log file, you would be presented with logs related to authentication and authorization on your Linux system.

Linux Log File Location auth

Similarly, the cron.log file displays information related to the cron service on your system.

However, as you can see from the diagram above, there is a coexistence of two different logging systems on your Linux server : rsyslog and systemd-journal.

Rsyslog and systemd-journal coexistence

Historically, a daemon was responsible for gathering logs from your applications on Linux.

On many old distributions, this task was assigned to a daemon called syslogd but it was replaced in recent distributions by the rsyslog daemon.

When systemd replaced the existing init process on recent distributions, it came with its own way of retrieving and storing logs : systemd-journal.

Now, the two systems are coexisting but their coexistence was thought to be backwards compatible with the ways logs used to be architectured in the past.

The main difference between rsyslog and systemd-journal is that rsyslog will persist logs into the log files available at /var/log while journald will not persist data unless configured to do it.

Journal Log Files Location

As you understood it from the last section, the systemd-journal utility also keeps track of logging activities on your system.

Some applications that are configured as services (an Apache HTTP Server for example) may talk directly to the systemd journal.

The systemd journal stores logs in a centralized way is the /run/log/journal directory.

The log files are stored as binary files by systemd, so you won’t be able to inspect the files using the usual cat or less commands.

Instead, you want to use the “journalctl” command in order to inspect log files created by systemd-journal.

$ journalctl

There are many different options that you can use with journalctl, but most of the time you want to stick with the “-r” and “-u” option.

In order to see the latest journal entries, use “journalctl” with the “-r” option.

$ journalctl -r

Journal Log Files Location journalctl-r

If you want to see logs related to a specific service, use the “-u” option and specify the name of the service.

$ journalctl -u <service>

For example, in order to see logs related to the SSH service, you would run the following command

$ journalctl -u ssh

Now that you have seen how you can read configuration files, let’s see how you can easily configure your logging utilities.

Linux Logging Configuration

As you probably understood from the previous sections, Linux logging is based on two important components : rsyslog and systemd-journal.

Each one of those utilities has its own configuration file and we are going to see in the following chapters how they can be configured.

Systemd journal configuration

The configuration files for the systemd journal are located in the /etc/systemd directory.

$ sudo vi /etc/systemd/journald.conf

The file named “journald.conf” is used in order to configure the journal daemon on recent distributions.

One of the most important options in the journal configuration is the “Storage” parameter.

As specific before, the journal files are not persisted on your system and they will be lost on the next restart.

To make your journal logs persistent, make sure to modify this parameter to “persistent” and to restart your systemd journal daemon.

Systemd journal configuration persistent

To restart the journal daemon, use the “systemctl” command with the “restart” option and specify the name of the service.

$ sudo systemctl restart systemd-journald

As a consequence, journal logs will be stored into the /var/log/journal directory next to the rsyslog log files.

$ ls -l /var/log/journal

Systemd journal configuration var-log-journal

If you are curious about the systemd-journal configuration, make sure to read the documentation provided by FreeDesktop.

Rsyslog configuration

On the other hand, the rsyslog service can be configured via the /etc/rsyslog.conf configuration file.

$ sudo vi /etc/rsyslog.conf

As specified before, rsyslog is essentially a Syslog collector but the main concept that you have to understand is that Rsyslog works with modules.

Rsyslog configuration rsyslog-card

Its modular architecture provides plugins such as native ways to transfer logs to a file, a shell, a database or sockets.

Working with rsyslog, there are two main sections that are worth your attention : modules and rules.

Rsyslog Modules

By default, two modules are enabled on your system : imuxsock (listening on the syslog socket) and imjournal (essentially forwarding journal logs to rsyslog).

Note : the imklog (responsible for gathering Kernel logs) might be also activated.

Rsyslog configuration modules-rsyslog

Rsyslog Rules

The rules section of rsyslog is probably the most important one.

On rsyslog, but you can find the same principles on old distributions with systemd, the rules section defines which log should be stored to your file system depending on their facility and priority.

As an example, let’s take the following rsyslog configuration file.

Rsyslog Rules rules-rsyslog

The first column describes the rules applied : on the left side of the dot, you define the facility and on the right side the severity.

Rsyslog Rules rsyslog-rules

A wildcard symbol “*” means that it is working for all severities.

As a consequence, if you want to tweak your logging configuration in order, say for example that for example you are interested in only specific severities, this is the file you would modify.

Linux Logs Monitoring Utilities

In the previous section, we have seen how you can easily configure your logging utilities, but what utilities can you use in order to read your Linux logs easily?

The easiest way to read and monitor your Linux logs is to use the tail command with the “-f” option for follow.

$ tail -f <file>

For example, in order to read the logs written in the auth.log file, you would run the following command.

$ tail -f /var/log/auth.log

Another great way of reading Linux logs is to use graphical applications if you are running a Linux desktop environment.

The “Logs” application is a graphical application designed in order to list application and system logs that may be stored in various logs files (either in rsyslog or journald).

Linux Logs Monitoring Utilities logs-application

Linux Logging Utilities

Now that you have seen how logging can be configured on a Linux system, let’s see a couple of utilities that you can use in case you want to log messages.

Using logger

The logger utility is probably one of the simpliest log client to use.

Logger is used in order to send log messages to the system log and it can be executed using the following syntax.

$ logger <options> <message>

Let’s say for example that you want to send an emergency message from the auth facility to your rsyslog utility, you would run the following command.

$ logger -p auth.emerg "Somebody tried to connect to the system"

Now if you were to inspect the /var/log/auth.log file, you would be able to find the message you just logged to the rsyslog server.

$ tail -n 10 /var/log/auth.log | grep --color connect>

Linux Logging Utilities var-log-auth

The logger is very useful when used in Bash scripts for example.

But what if you wanted to log files using the systemd-journal?

Using systemd-cat

In order to send messages to the systemd journal, you have to use the “systemd-cat” command and specify the command that you want to run.

$ systemd-cat <command> <arguments>

If you want to send the output of the “ls -l” command to the journal, you would write the following command

$ systemd-cat ls -l

Using systemd-cat journalctl-2

It is also possible to send “plain text” logs to the journal by piping the echo command to the systemd-cat utility.

$ echo "This is a message to journald" | systemd-cat

Using wall

The wall command is not related directly to logging utilities but it can be quite useful for Linux system administration.

The wall command is used in order to send messages to all logged-in users.

$ wall -n <message>

If you were for example to write a message to all logged-in users to notify them about the next server reboot, you would run the following command.

$ wall -n "Server reboot in five minutes, close all important applications"

Using wall wall-message

Conclusion

In this tutorial, you learnt more about Linux logging : how it is architectured and how different logging components (namely rsyslog and journald) interact together.

You learnt more about the Syslog protocol and how collectors can be configured in order to log specific events on your system.

Linux logging is a wide topic and there are many more topics for you to explore on the subject.

Did you know that you can build centralized logging systems in order to monitor logs on multiple machines?

If you are interested about centralized logging, make sure to read our guide!

Also, if you are passionate about Linux system administration, we have a complete section dedicated to it on the website, so make sure to check it out!