 |
At the heart of every modern Mac and Linux computer is the “terminal.� The terminal evolved from the text-based computer terminals of the 1960s and ’70s, which themselves replaced punch cards as the main way to interact with a computer. It’s also known as the command shell, or simply “shell.� Windows has one, too, but it’s called the “command prompt� and is descended from the MS-DOS of the 1980s.
Mac, Linux and Windows computers today are mainly controlled through user-friendly feature-rich graphical user interfaces (GUIs), with menus, scroll bars and drag-and-drop interfaces. But all of the basic stuff can still be accomplished by typing text commands into the terminal or command prompt.
Using Finder or Explorer to open a folder is akin to the cd command (for “change directory�). Viewing the contents of a folder is like ls (short for “list,� or dir in Microsoft’s command prompt). And there are hundreds more for moving files, editing files, launching applications, manipulating images, backing up and restoring stuff, and much more.
So, why would anyone want to bother with these text commands when you can use the mouse instead? The main reason is that they are very useful for controlling remote computers on which a GUI is not available, particularly Web servers, and especially Linux Web servers that have been stripped of all unnecessary graphical software.
Sometimes these lean Linux servers are managed through a Web browser interface, such as cPanel or Plesk, letting you create databases, email addresses and websites; but sometimes that is not enough. This article provides a broad introduction to text commands and the situations in which they are useful. We’ll cover the following:
- Why knowing a few commands is useful;
- Issuing commands on your own computer;
- Using SSH to log into your Web server;
- Getting your bearings:
pwd,cs ls; - Viewing and moving files:
cat,more,head,tail,mv,cp,rm; - Searching for files:
find; - Looking through and editing files:
grep,vi; - Backing up and restoring files and databases:
tar,zip,unzip,mysqldump,mysql; - File permissions:
chmod.
Why Knowing A Few Linux Commands Is Useful
As a website developer or server administrator, you would gain a big asset in becoming comfortable with these commands: for website emergencies, to configure a server and for your CV. It can also save you money. Many hosting companies offer fully managed servers but at a high monthly premium. Or else they charge by the hour for technical support.
Perhaps you need to archive some big files or make a change to the httpd.conf file or figure out why your website’s images have suddenly stopped loading. You might not want to pay $50 to your server’s administrator for a five-minute job. This article gives you the tools to make such changes yourself.
And why “Linux� commands? Two main types of servers are available today: Windows and UNIX. UNIX-based servers include Linux (which split off in 1991), Mac OS X (2002) and several more traditional UNIX systems, such as BSD, Solaris and HP-UX. Linux commands are basically UNIX commands and so will run on all of them. In fact, I use the term “Linux� here only because it is more common and less frightening than “UNIX.� Windows servers, on the other hand, have a much smaller market share and are more often controlled through GUIs, such as Remote Desktop and VNC, rather than the command line.
In fact, a November 2011 survey showed that Apache accounted for about 65% of all Web servers. Apache usually runs in the popular LAMP configuration: Linux, Apache, MySQL and PHP. Microsoft was a distant second, with 15%. Third place nginx runs on Linux, UNIX, Mac and Windows. So, the commands in this article will work on at least two thirds of all servers.
Issuing Commands To Your Own Computer
You can quickly experiment with text commands on your own computer. On Mac with OS X, go to Applications → Utilities, and run Terminal. On a PC with Windows, go to Start → All Programs → Accessories, and choose “Command Prompt.� On Ubuntu Linux, go to Applications → Accessories, and choose Terminal.
On Windows you should see this:
The Windows command prompt
This is the command line (i.e. shell, prompt or terminal) on your own computer. You can type dir on Windows or ls on Linux or Mac followed by “Enter� to see a list of the files in the current “directory� (i.e. folder, location or path).
All we will be doing for the rest of this article is opening up one of these terminals on a remote computer: your Web server.
You may have used VNC or Remote Desktop, which allow you to actually view the desktop on someone else’s computer: your screen shows their screen, your mouse controls their mouse, your keyboard mimics their keyboard.
The terminal is similar to this but without the fancy menus or scroll bars. If you were to plug a keyboard and screen into your Web server, sitting in a fireproof basement somewhere, you would probably see one of these terminals, waiting patiently for your user name and password.
Using SSH To Log Into Your Web Server
The application SSH, or Secure Shell, is used to log into Web servers. It often takes the same user name and password as FTP, but it has to be allowed by your host. If you have a dedicated Web server, it is probably already allowed. If you use cloud hosting, then you might need to request it first. If you are on shared hosting, you’ll definitely need to request it, and the administrator may refuse.
On Linux or Mac, open up Terminal as described above and type the following:
ssh -l username www.myserver.com
The -l stands for “log in as,� and your user name goes after it. If SSH is allowed, then it will ask for a password. If not, you’ll get an error message, like this one:
Running the ssh command and being denied access
On Windows, you will need to download some SSH software. Putty is a popular and easy choice. It downloads as a single EXE file, which you can save to your desktop and run right away. Type your website as the host name, check the SSH box under “Connection Type,� and click “Open.� It will ask for your user name and then your password.
Running Putty on Windows in order to SSH to your Web server
Once successfully logged in, you will usually see a welcome message. After that, you will be presented with a few letters and a $ sign (or a # sign if you have logged in as root). The letters often represent your user name and where you’ve come from, or the name of the server. A ~ indicates that you are in your home directory. The $ is the prompt; it indicates that you can start typing commands now, something like:
A successful SSH log-in to a Web server. The $ means we can start typing commands.
The next section introduces a few basic commands.
Getting Your Bearings
On Windows, when you go to “My Documents� from the Start menu, it opens your “My Documents� directory in Windows Explorer and shows the contents. If some nosy colleague walked by and asked “What directory are you in?� you could say “I’m in my documents.�
If you SSH’ed to a server as the user “admin,� you would land in admin’s home directory, probably /home/admin. You can verify this by typing the command pwd, which shows your current location (i.e. folder, directory or path).
The pwd command tells you where you are, cd changes the directory and ls shows the contents of a directory.
To change to another directory, use the cd command with the destination, like so:
cd /
This will change the directory to /, the top of the whole UNIX directory structure. The command ls lists the contents of the current directory, in this case /.
In the screenshot above, the terminal is color-coded. Dark-blue entries are subdirectories, and black entries are files. A lot of the interesting stuff on Web servers happens in the /etc, /home and /var directories. Using just cd and ls, you can explore your server and find out where stuff is.
When using cd, you can specify the new directory absolutely (beginning with a slash, like /var/www) or relative to your current location (without the initial slash). You can also go up a directory with two dots. Practice with the sequence below, pressing “Enter� after each command. Can you guess what the last command will tell you?
cd /var ls cd www ls cd .. pwd
Viewing And Moving Files
On many Linux servers, websites are located in /var/www/vhosts. You can check on your server by doing the following:
cd /var/www/vhosts ls
If you see a list of websites, you can move into one of them. Within the website’s main directory, you will probably see the same files that you see when you FTP to the website, things such as httpdocs (where your website’s files are), httpsdocs (if you have a separate secure website), conf (configuration files), statistics (logs and compiled statistics), error_docs, private and more.
You can then change into your website’s public-facing directory, which is myserver.com/httpdocs in this example:
cd myserver.com ls cd httpdocs ls
Now you have arrived, and you can run a new command, cat, which displays the contents of a file. For instance, if you have an index.html file, run:
cat index.html
If your index.html file is more than a few lines long, it will rush past in a blur. You can use the more command to show it slowly, one page at time. After you type the command below, it will show you the first page. You can press the space bar to show the next page, “Enter� to show the next line, and Q to quit.
more index.html
You can also show just the first few or last few lines of a file with the head and tail commands. It shows 10 lines by default, but you can pass in any number:
head index.html tail -20 index.html
If you would like to rename this file, use the mv command, short for “move�:
mv index.html indexold.html
Similarly, the cp is the copy command, and rm removes files.
cp index.html indexold.html rm indexold.html
Below is a string of commands in action. In order, it confirms the current directory with pwd, looks at the contents with ls, views index.html with cat, then renames it with mv, and finally removes it with rm, with a lot of ls in between to show the changes.
The cat, mv and rm commands in action, for displaying, moving and then removing a file.
More Advanced Tip: Changing the Prompt
Note that in our initial examples, the full prompt included the current directory. For instance, in [admin@myserver /]$, the / indicated that the user was in the / directory. In the example directly above, it was removed, or else it would have crowded the screenshot by constantly saying [admin@myserver /var/www/vhosts/myserver.com/httpdocs]$.
You can change the prompt to whatever you want by setting the PS1 environment variable. Here are a couple of examples, the latter including the user, host and current directory:
PS1="[woohoooo ]$ "
PS1='[${USER}@${HOSTNAME} ${PWD}]$ '
Searching For Files
On big websites, files can get lost. Perhaps you vaguely remember uploading a new version of your client’s logo about four months ago, but it has since fallen out of favor and been replaced. Now, out of the blue, the client wants it back. You could download everything from the server using FTP and search the files using Finder or Explorer. Or you could log in and search using the command line.
The find command can search through files by name, size and modified time. If you just give it a directory, it will list everything that the directory contains. Try this:
find /var/www
You will probably see lots and lots of file names whizzing past. If you have many websites, it could continue for a couple of minutes. You can stop it by hitting Control + C (i.e. holding down the Control key on your keyboard and pressing the letter C). That’s the way to interrupt a Linux command. A more useful command would be:
find /var/www | more
The pipe symbol (|) takes the output of one command (in this case, the long list of files produced by find) and passes it to another command (in this case, more, which shows you one page of files at a time). As above, press the space bar to show the next page, and Q to quit.
To search for a specific file name, add -name and the file name. You can use \* as a wild card (the backslash is not always necessary but is good practice with the find command). You can combine searches using -o (for “or�). If you leave out the -o, it becomes an “and.�
find /var/www -name logo.gif find /var/www -name \*.gif find /var/www -name \*.gif -o -name \*.jpg
You can also search by size by adding -size. So, you could look for all GIFs between 5 and 10 KB:
find /var/www -name \*.gif -size +5k -size -10k
Similarly, to find a file that was last changed between 90 and 180 days ago, you can use -ctime:
find /var/www -name \*.gif -ctime +90 -ctime -180
In both of these cases, you will probably also want to know the actual file size and date last changed. For this, you can add -printf, which is similar to the C function printf in that you use the % sign to output various information. This command outputs the file size (up to 15 characters wide), the date and time changed (down to the nanosecond) and the file name:
find /var/www -name \*.gif -size +5k -size -10k -ctime +90 -ctime -180 -printf "%10s %c %p\n"
With that whopper, you have hopefully found the missing file. Here is an example:
Searching for all GIFs within a single website, and displaying the file sizes, changed times and file names.
Another useful parameter is -cmin, which lets you see files that have changed in the last few minutes. So, if something goes wrong on a website, you can run this to see everything that has changed in the last 10 minutes:
find /var/www -cmin -10 -printf "%c %p\n"
This will show files and directories that have changed. Thus, it won’t show files that have been removed (because they are no longer there), but it will show the directories that they were removed from. To show only files, add -type f:
find /var/www -cmin -10 -type f -printf "%c %p\n"
More Advanced Tip: Reading the Manual
I didn’t have to remember all of the variations above. I consulted the manual several times, like so:
man find
While reading a manual page, the controls are the same as more: space bar for paging, “Enter� to go forward one line and Q to quit. The up and down arrows also work. You can search within a page of the manual by typing / and a keyword, such as /printf. This will jump you to the next occurrence of that term. You can search backwards with ?printf, and you can repeat the search by pressing N.
Looking Through And Editing Files
Most visual code editors allow you to search through many files when you’re looking for a particular variable or bit of HTML. You can also do this directly on the server using the command grep. This is useful when something goes wrong on a complex website with hundreds of files and you have to find the error and fix it fast.
Let’s say you view the HTML source and see that the error happens right after <div id="left">. You can let grep do the searching for you. Give it the thing to be searched for and the files to search in. These commands change to the website directory and grep through all files ending in php. You need to put quotes around the HTML because it contains spaces, and the inner quotes have to be escaped with backslashes:
cd /var/www/vhosts/myserver.com/httpdocs/ grep "<div id=\"left\">" *.php
This will tell you which files in the current directory contain that bit of HTML. If you want to search in subdirectories, you can use the -r option with a directory at the end, instead of a list of files. The single dot tells it to start in the current directory.
grep -r "<div id=\"left\">" .
Alternatively, you could use the find command from above to tell it which files to look in. To put a command within a command, enclose it in back apostrophes. The following searches only for the HTML in PHP files modified in the last 14 days:
grep "<div id=\"left\">" `find . -name \*.php -ctime -14`
You can also add -n to show the line numbers, as in this example:
Searching for a bit of HTML within the PHP files in the current directory
And how do you quickly fix an error when you find it? To do that, you will need to start up a Linux text editor. Different editors are available, such as pico and emacs, but the one that is guaranteed to be there is vi. To edit a file, type vi and the file name:
vi index.php
vi is a complex editor. It can do most of the amazing things that a fully featured visual editor can do, but without the mouse. In brief, you can use the arrow keys to get around the file (or H, J, K and L on very basic terminals where even the arrow keys don’t work). To delete a character, press X. To delete a whole line, press DD. To insert a new character, press I. This takes you into “insert mode,� and you can start typing. Press the Escape key when finished to go back to “command mode.� Within command mode, type :w to save (i.e. write) the file and :q to quit, or :wq to do both at the same time.
The vi editor also supports copying and pasting, undoing and redoing, searching and replacing, opening multiple files and copying between them, etc. To find out how, look for a good vi tutorial (such as “Mastering the VI Editor�). Note also that on many computers, vi is just a shortcut to vim, which stands for “vi improved,� so you can follow vim tutorials, too.
Editing files with the vi text editor
More Advanced Tip: Tab Completion
When changing directories and editing files, you might get tired of having to type the file names in full over and over again. The Terminal loses some of its shine this way. This can be avoided with command-line completion, performed using tabs.
It works like this: start typing the name of a file or a command, and then press Tab. If there is only one possibility, Linux will fill in as much as it can. If nothing happens, it means there is more than one possibility. Press Tab again to show all of the possibilities.
For example, if above I had typed…
vi i
… And then pressed Tab, it would have filled in the rest for me…
vi index.php
… Unless several files started with I. In that case, I would have had to press Tab again to see the options.
Backing Up And Restoring Files And Databases
Some Linux servers do support the zip command, but all of them support tar, whose original purpose was to archive data to magnetic tapes. To back up a directory, specify the backup file name and the directory to back up, such as:
cd /var/www/vhosts/myserver.com/httpdocs/ tar czf /tmp/backup.tgz .
The czf means “create zipped file.� The single dot stands for the current directory. You can also back up individual files. To back up just things changed in the last day, add the find command:
tar cfz /tmp/backup.tgz `find . -type f -ctime -1`
Both of these commands put the actual backup file in the temporary /tmp directory — if the backup file is in the same directory that you are backing up, it will cause an error. You can move the file to where you need it afterwards. To see what is in an archive, use the tzf options instead:
tar tfz /tmp/backup.tgz
Creating and showing the contents of a backup file using tar
To restore things, use xzf, for “extract from zipped file.� First, run a listing as above to check what’s in there, and then restore one or more of the files. The second command restores all of the files from the archive into the current directory:
tar xfz /tmp/backup.tgz ./index.php ./test.php tar xfz /tmp/backup.tgz
If your server has the zip command, then run these commands to do the same thing:
cd /var/www/vhosts/myserver.com/httpdocs/ zip -r /tmp/backup.zip . zip -r /tmp/backup.zip `find . -type f -ctime -1` unzip -l /tmp/backup.zip unzip /tmp/backup.zip test.php unzip /tmp/backup.zip
If your Web server uses MySQL, then you might want to regularly back up your data. For this, there is the mysqldump command. The format of the command is:
mysqldump --user="username" --password="password" --add-drop-table database
Replace the user name, password and database with your values. Instead of specifying a database, you can use -A to dump all databases. If you get errors about table locking, you can add --single-transaction. Once you submit the user name and password, this will output a load of SQL in a long blur. To save the output to a file, you will need to use the > symbol. This sends the output of a command to a file.
mysqldump --user="username" --password="password" --add-drop-table database > /tmp/db.sql
To restore a database backup, you can use the mysql command. This command lets you run SQL statements from the command line. For example, the following command gets you into the database:
mysql --user="username" --password="password" dbname
At the mysql> prompt, you can type an SQL statement such as:
mysql> SHOW TABLES; mysql> SELECT * FROM customers;
For restoring, you’ll need to use the pipe (|), which will send the output from one command into another. In this case, cat will output the database backup file and send it into the mysql command:
cat /tmp/db.sql | mysql --user="username" --password="password" dbname
If people are looking over your shoulder while you’re doing this, you might not want to type the password directly into the command. In this case, just leave it out, and mysql or mysqldump will ask for it instead.
cat /tmp/db.sql | mysql --user="username" --password dbname
Once you’ve created the database backup file, you can include it in the backups we did above:
tar czf /tmp/backup.tgz . /tmp/db.sql
More Advanced Tip: Hidden Files and Wildcards
Many websites use a file called .htaccess to implement URL rewriting and password protection. In UNIX, all files starting with a single dot are hidden. They won’t show up when you do ls, and they won’t get backed up if you do this:
tar czf /tmp/backup.tgz *
The * is a wildcard. Before the command executes, the * is replaced with all non-hidden files in the current directory. To include hidden files as well, it’s better to back up the whole directory as above using a single dot:
tar czf /tmp/backup.tgz .
To show hidden files when doing a directory listing, add -a to the command:
ls -a ls -la
File Permissions
If you use FTP regularly to upload files to websites, then you might be familiar with permissions. All files and directories on Linux (and Mac, Windows and other UNIX systems) have an owner, a group and a set of flags specifying who can read, write and execute them.
The list of user names (and, thus, potential file owners) on a UNIX system is stored in the file /etc/passwd. You can try:
more /etc/passwd
The Apache Web server is started by a command when the Web server boots up. But the user who starts Apache is often a restricted and unprivileged user, such as nobody or apache or www-data. This is for security reasons, to prevent someone from hacking into the website and then gaining control of the whole server. You can find out who that user is by running the command below and looking in the first column. The ps aux command shows all of the processes running on the server, and grep shows only processes that contain the word “apache.�
ps aux | grep apache
This can cause conflicts, though. If you upload a file to a website via FTP and log in as admin, then the file will be owned by admin. If Apache was started by the user named nobody, then Apache might not be able to read that file and won’t be able to send it to any users who request it when viewing the website. Instead, users will see a broken image or a message such as “403 Forbidden. You don’t have permission to access that file.�
A subtler and more common problem is when an image can be viewed but not overwritten or removed via the website’s content management system (CMS). In this case, the user nobody can read the file but can’t write to it.
You can view permissions using the ls command with an extra -l, like so:
ls -l
The command ls -l shows information about permissions, owners, size and date.
This directory contains three files, with three subdirectories shown in green. The first letter on each line indicates the type: d for directory and - for normal file. The next nine letters are the permissions; they indicate the read, write and execute permissions for the owner, group and everyone else. After the number (which represents the size) is the owner and group for the file. These files are all owned by admin. This is followed by the file size (less useful for directories) and the date and time of the last modification.
Below is another example of three files in an images subdirectory. Two of the files were uploaded by admin via FTP, and Apache was started by the user www-data. One of the files will be unviewable through a Web browser. Which do you think it is?
The answer is bg.jpg. Both bg.jpg and logo2.gif have the same permissions: only the owner can read and write them. The logo2.gif file is OK because the owner is www-data, so that file can be accessed, read and returned by Apache. The logo.gif file is also OK because it has r in all three permissions (i.e. owner, group and everyone else). But bg.jpg will fail because only the user admin can read it, not the user who started Apache. If you were to access that file in a Web browser, you would see something like this:
What happens when you try to access a file without the correct permissions in a browser.
These sorts of errors can be resolved with the chmod command, which changes file permissions. The three sets of permissions are represented in commands with u (“user� or owner), g (“group�), o (“other� or everyone else) or a (“all�). So, to enable all users to read bg.jpg, either of these would work:
chmod go+r images/bg.jpg chmod a+r images/bg.jpg
If this file were also part of a CMS, then you’d have to also add write permissions before the CMS could overwrite it:
chmod a+rw images/bg.jpg
You can also make these changes to all files in all of the subdirectories by adding -R. This recursive operation is not supported by some FTP programs and so is a useful command-line tool:
chmod -R a+rw images/
Directories also need the x (“execute� permission), but files generally don’t (unless they are in a cgi-bin). So, you can give everything rwx (read, write and execute) permissions and then take away the x from the files:
chmod -R a+rwx images/ chmod -R a-x `find images/ -type f`
However, this does leave everything rather open, making it easier for hackers to gain a foothold. Ideally, your set of permissions should be as restrictive as possible. Files should be writable by the Apache user only when needed by the CMS.
More Advanced Tip: Chown and the Superuser
Another useful permissions command is chown. It changes the owner of a file. However, you have to be logged in as a user with sufficient privileges (such as root) in order to run it. To make www-data the owner of bg.jpg, run this:
chown www-data images/bg.jpg
This will probably return “Permission denied.� You have to run the command as the superuser. For this, you will need to find the root password for your server, and then run the following:
sudo chown www-data images/bg.jpg
You will definitely need to be the superuser if you want to edit configuration files, such as Apache’s:
sudo vi /etc/httpd/conf/httpd.conf
If you want to become the superuser for every command, run this:
su
This is dangerous, though, because you could easily accidentally remove things — especially if you are using the rm command, and particularly if you’re using it in recursive mode (rm -r), and most especially if you also force the changes and ignore any warnings (rm -r -f).
Conclusion
This article has introduced some very useful Linux commands, a potential asset for any aspiring Web worker and a surefire way to impress a dinner date.
For a few more commands related specifically to website crashes, check out the Smashing Magazine article “What to Do When Your Website Goes Down.� For a broader view, try this list of Linux commands. And the “Mastering the VI Editor� tutorial mentioned above explains vi well.
Hopefully, you now have the tools and confidence to pitch in the next time one of your websites has a problem.
(al)
© Paul Tero for Smashing Magazine, 2012.
