Introduction to Generic Netlink, or How to Talk with the Linux Kernel

2023-02-10T22:38:08+00:00

Are you writing some code in kernel-space and want to communicate with it from the comfort of user-space? You are not writing kernel code, but want to talk with the Linux kernel? Serve yourself some good java (the good hot kind, not the Oracle one), make yourself comfortable and read ahead!

I recently got myself into programming in the Linux Kernel, more specifically kernel space modules, and one of the APIs that I had to study was Generic Netlink. As is usual with most Linux Kernel APIs the documentation, outside of sometimes fairly well commented code, is a bit lacking and/or old.

Hence why I decided to make a little example for myself on how to use Generic Netlink for kernel-user space communications, and an introductory guide using the aforementioned example for my colleagues and any other person interested in using Generic Netlink but not knowing where to start.

This guide covers the following:

Registering Generic Netlink families in kernel.</li>
Registering Generic Netlink operations and handling them in kernel.</li>
Registering Generic Netlink multicast groups in kernel.</li>
Sending "events" through Generic Netlink from kernel.</li>
Connecting to Generic Netlink from a user program.</li>
Resolving Generic Netlink families and multicast groups from a user program.</li>
Sending a message to a Generic Netlink family from a user program.</li>
Subscribing to a Generic Netlink multicast group from a user program.</li>
Listening for Generic Netlink messages from a user program.</li> </ul>
What is Netlink?</h2>
Netlink is a socket domain created with the task of providing IPC for the Linux Kernel, especially kernel<->user IPC. Netlink was created initially with the intention of replacing the aging ioctl() interface, by providing a more flexible way of communicating between kernel and user programs.
Netlink communication happens over standard sockets using the AF_NETLINK</code> domain. Nonetheless, on the user land side of things, libraries exist that provide a more convenient way of using the Netlink interface, such as libnl^{1</a>.}
That said, new Netlink families aren't being created anymore and new code doesn't use Netlink directly. The classic use of Netlink is relegated to already existing families such as NETLINK_ROUTE</code>, NETLINK_SELINUX</code>, etc. The main problem with Netlink is that it uses static allocation of IDs which are limited to 32 unique families, which greatly limits its users and may cause conflicts with out-of-tree modules.
Presenting Generic Netlink</h2> Generic Netlink was created to fix the deficiencies of Netlink as well as bringing some quality of life improvements. It is not a separate socket domain though, it's more of an extension of Netlink. In fact, it is a Netlink family — NETLINK_GENERIC</code>. Generic Netlink has been around since 2005 so it is a well established interface for kernel<->userspace IPC. Some notable users of Generic Netlink include subsystems such as 802.11, ACPI, Wireguard, among others. The main features that Generic Netlink brings to the table are dynamic family registration, introspection and a simplified kernel API. This tutorial is focused specifically on Generic Netlink, since it's the standard way of communicating with the kernel in ways that are more sophisticated than a simple sysfs file. Generic Netlink bus diagram </figcaption> Some theory</h2> As I've already mentioned, Netlink works over the usual BSD sockets. A Netlink message always starts with a Netlink header, followed by a protocol header, that is in the case of Generic Netlink, the Generic Netlink header. Netlink messages</h3> The headers look like this: Netlink Header </figcaption> Generic Netlink Header </figcaption> Or as described by the following C structures: struct nlmsghdr { __u32 nlmsg_len; __u16 nlmsg_type; __u16 nlmsg_flags; __u32 nlmsg_seq; __u32 nlmsg_pid; }; struct genlmsghdr { __u8 cmd; __u8 version; __u16 reserved; }; </code></pre> Netlink header fields meaning: Length — the length of the whole message, including headers.</li> Type — the Netlink family ID, in our case Generic Netlink.</li> Flags — a do or dump; more on that later.</li> Sequence — sequence number; also more on that later.</li> Port ID — set to 0, since we are sending from kernel.</li> </ul> and for Generic Netlink: Command: operation identifier as defined by the Generic Netlink family.</li> Version: the version of the Generic Netlink family protocol.</li> Reserved: as its name implies ¯\(ツ)/¯.</li> </ul> Most of the fields are pretty straight forward, and the header is not usually filled manually by the Netlink user. Some of the information contained in the headers is provided by the user through the API when calling the different functions. Some of that information are things like the flags and sequence numbers. There are three types of message operations that are usually performed over a Netlink socket: A do operation</li> A dump operation</li> And multicast messages, or asynchronous notifications.</li> </ul> There are many different ways of sending messages over Netlink, but these are the most used in Generic Netlink. A do operation is a single action kind of operation in which the user program sends the message and receives a reply that could be an acknowledgment or error message, or maybe even a message with some information. A dump operation is one for (duh) dumping information, usually more than fits in one message. The user program also sends a message but receives multiple reply message until received a NLMSG_DONE</code> message that signals the end of the dump. Whether an operation is a do or a dump is set using the flags field: NLM_F_REQUEST | NLM_F_ACK</code> for do.</li> NLM_F_REQUEST | NLM_F_ACK | NLM_F_DUMP</code> for dump.</li> </ul> Now the third type, the multicast messages, are used for sending notifications to the users that are subscribed to them via the generic netlink multicast group. As we saw, there's also a sequence number field in the Netlink header. However, unlike in other protocols, Netlink doesn't manipulate or enforce the sequence number itself. It's provided as a way to help keep track of messages and replies. In practice, the user program would increase the sequence number with each message sent, and the kernel module would send the reply(ies) with the same sequence as the command message. Multicast message are usually sent with a sequence number of 0. Message payload</h3> Netlink provides a system of attributes to encode data with information such as type and length. The use of attributes allows for validation of data and for a supposedly easy way to extend protocols without breaking backward compatibility. You can also encode your own types, such as a struct in a single attribute. However, the use of Netlink attributes for each field is encouraged. The attributes are encoded in LTV format and are padded such that each attribute begins at an offset that is a multiple of 4 bytes. The length fields, both in the message header and attribute, always include the header, but not the padding. Netlink attribute diagram </figcaption> The attributes in a Netlink, and hence Generic Netlink, message are not necessarily added always in the same order, which is why they should be walked and parsed. Netlink provides for a way to validate that a message is correctly formatted using so called "attribute validation policies", represented by struct nla_policy</code>. I do find it a bit strange that the structure is not exposed to user space and hence validation seems to be performed by default only on the kernel side. It can also be done on user space and libnl also provides its own struct nla_policy</code> but it differs from the kernel one which means you basically have to do duplicate work to validate the attributes on both sides. The types of messages and operations are defined by the so called Netlink commands. A command correlates to one message type, and also might correlate to one op(eration). Each command or message type can have or use one or more attributes. Families</h3> I have already mentioned families in this text and in different contexts. Unfortunately, as is common in the world of computer programming, things sometimes aren't named in the very best way hence we end up with situations like this one. Sockets have families, of which we use the AF_NETLINK</code>. Netlink also has families, of which there are only 32, and no more are planned or should be introduced to the Linux kernel; the family we use is NETLINK_GENERIC</code>. Last but not least, Generic Netlink also has families, although these are dynamically registered and a whopping total of 1024 can be registered at a single time. Generic Netlink families are identified by a string, such as "nl80211", for example. Since the families are registered dynamically, that means that their ID can change from one computer to another or even one boot to another, so we need to resolve them before we can send messages to a family. Generic Netlink in itself provides a single statically allocated family called nlctrl</code> which provides with a command to resolve said families. It also provides since not long ago a way for introspecting operations and exposing policies to user space, but we are not going to go into detail on this in this tutorial. One more thing of note is that a single Generic Netlink socket is not bound to any one family. A Generic Netlink socket can talk to any family at any time, it just needs to provide the family ID when sending the message, by using the type field as we saw earlier. Multicast groups</h3> There are some message that we would like to send asynchronously to user programs in order to notify them of some events, or just communicate information as it becomes available. This is where multicast groups come in. Generic Netlink multicast groups, just like families, are dynamically registered with a string name and receiving a numeric ID upon registration. In other words, they must also be resolved before being to subscribe to them. Once subscribed to a multicast group, the user program will receive all message sent to the group. In order to avoid mixing sequence numbers with unicast messages and to make handling easier, it is recommended to use a different socket for multicast messages. Getting our hands dirty</h2> There's much more about Generic Netlink and especially classic Netlink, but those were the most important concepts to know about when working with Generic Netlink. That said, it's not very interesting just knowing about something, we are here for the action after all. I have made an example of using Generic Netlink that consists of two parts. A kernel module, and a userland program. The kernel module provides a single generic netlink operation and a multicast group. The message structure is the same for the do op and the multicast notification. The first reads a string message, prints it to the kernel log, and sends its own message back; and the second sends a notification upon reading a message from sysfs, echoing it. The user space program connects to Generic Netlink, subscribes to the multicast group, sends a message to our family and prints out the received messages. I'll be explaining them step by step with code listings in this article. The full source code for both parts can be found at https://git.yaroslavps.com/genltest/</a>. The land of the Kernel</h3> Using Generic Netlink from kernel space is pretty straightforward. All we need to start using it is to include a single header in our file, net/genetlink.h</code>. In total all the headers that we need to start working with our example are as follows: #include <linux/module.h> #include <net/genetlink.h> </code></pre> We'll need some definitions and enumerations that will be shared between kernel space and user space, we'll put them in a header file that we'll call genltest.h</code>: #define GENLTEST_GENL_NAME "genltest" #define GENLTEST_GENL_VERSION 1 #define GENLTEST_MC_GRP_NAME "mcgrp" /* Attributes */ enum genltest_attrs { GENLTEST_A_UNSPEC, GENLTEST_A_MSG, __GENLTEST_A_MAX, }; #define GENLTEST_A_MAX (__GENLTEST_A_MAX - 1) /* Commands */ enum genltest_cmds { GENLTEST_CMD_UNSPEC, GENLTEST_CMD_ECHO, __GENLTEST_CMD_MAX, }; #define GENLTEST_CMD_MAX (__GENLTEST_CMD_MAX - 1) </code></pre> There we defined the name of our family, our protocol version, our multicast group name, the attributes that we will use in our messages and our commands. Back in our kernel code, we make a validation policy for our "echo" command: /* Attribute validation policy for our echo command */ static struct nla_policy echo_pol[GENLTEST_A_MAX + 1] = { [GENLTEST_A_MSG] = { .type = NLA_NUL_STRING }, }; </code></pre> Make an array with our Generic Netlink operations: /* Operations for our Generic Netlink family */ static struct genl_ops genl_ops[] = { { .cmd = GENLTEST_CMD_ECHO, .policy = echo_pol, .doit = echo_doit, }, }; </code></pre> Similarly an array with our multicast groups: /* Multicast groups for our family */ static const struct genl_multicast_group genl_mcgrps[] = { { .name = GENLTEST_MC_GRP_NAME }, }; </code></pre> Finally the struct describing our family, where we include everything so far: /* Generic Netlink family */ static struct genl_family genl_fam = { .name = GENLTEST_GENL_NAME, .version = GENLTEST_GENL_VERSION, .maxattr = GENLTEST_A_MAX, .ops = genl_ops, .n_ops = ARRAY_SIZE(genl_ops), .mcgrps = genl_mcgrps, .n_mcgrps = ARRAY_SIZE(genl_mcgrps), }; </code></pre> On initialization of our module, we need to register our family with Generic Netlink. For that we just need to pass it our genl_family</code> structure: ret = genl_register_family(&genl_fam); if (unlikely(ret)) { pr_crit("failed to register generic netlink family\n"); // etc... } </code></pre> And similarly, on module exit we need to unregister it: if (unlikely(genl_unregister_family(&genl_fam))) { pr_err("failed to unregister generic netlink family\n"); } </code></pre> As you may have noticed, we set our doit callback for our "echo" command to a echo_doit</code> function. Here's what it looks like: /* Handler for GENLTEST_CMD_ECHO messages received */ static int echo_doit(struct sk_buff *skb, struct genl_info *info) { int ret = 0; void *hdr; struct sk_buff *msg; /* Check if the attribute is present and print it */ if (info->attrs[GENLTEST_A_MSG]) { char *str = nla_data(info->attrs[GENLTEST_A_MSG]); pr_info("message received: %s\n", str); } else { pr_info("empty message received\n"); } /* Allocate a new buffer for the reply */ msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL); if (!msg) { pr_err("failed to allocate message buffer\n"); return -ENOMEM; } /* Put the Generic Netlink header */ hdr = genlmsg_put(msg, info->snd_portid, info->snd_seq, &genl_fam, 0, GENLTEST_CMD_ECHO); if (!hdr) { pr_err("failed to create genetlink header\n"); nlmsg_free(msg); return -EMSGSIZE; } /* And the message */ if ((ret = nla_put_string(msg, GENLTEST_A_MSG, "Hello from Kernel Space, Netlink!"))) { pr_err("failed to create message string\n"); genlmsg_cancel(msg, hdr); nlmsg_free(msg); goto out; } /* Finalize the message and send it */ genlmsg_end(msg, hdr); ret = genlmsg_reply(msg, info); pr_info("reply sent\n"); out: return ret; } </code></pre> In summary, when handling a do command we follow these steps: Get the data from the incoming message from the genl_info</code> structure.</li> Allocate a new message buffer for the reply.</li> Put the Generic Netlink header in the message buffer; notice that we use the same port id and sequence number as in the incoming message since this is a reply.</li> Put all our payload attributes.</li> Send the reply.</li> </ol> Now let's take a look at how to send multicast notifications. I've used sysfs to make this example a little bit more fun; I've created a kobj called genltest</code> which contains a ping</code> attribute from which we will echo what is written to it. For brevity I'll elide the sysfs code from the article and just add the function that forms and sends the message here: /* Multicast ping message to our genl multicast group */ static int echo_ping(const char *buf, size_t cnt) { int ret = 0; void *hdr; /* Allocate message buffer */ struct sk_buff *skb = genlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL); if (unlikely(!skb)) { pr_err("failed to allocate memory for genl message\n"); return -ENOMEM; } /* Put the Generic Netlink header */ hdr = genlmsg_put(skb, 0, 0, &genl_fam, 0, GENLTEST_CMD_ECHO); if (unlikely(!hdr)) { pr_err("failed to allocate memory for genl header\n"); nlmsg_free(skb); return -ENOMEM; } /* And the message */ if ((ret = nla_put_string(skb, GENLTEST_A_MSG, buf))) { pr_err("unable to create message string\n"); genlmsg_cancel(skb, hdr); nlmsg_free(skb); return ret; } /* Finalize the message */ genlmsg_end(skb, hdr); /* Send it over multicast to the 0-th mc group in our array. */ ret = genlmsg_multicast(&genl_fam, skb, 0, 0, GFP_KERNEL); if (ret == -ESRCH) { pr_warn("multicast message sent, but nobody was listening...\n"); } else if (ret) { pr_err("failed to send multicast genl message\n"); } else { pr_info("multicast message sent\n"); } return ret; } </code></pre> The process is very similar to the do operation, except that we are not responding to a request but sending an asynchronous message. Because of that we are setting the sequence number to 0, since it is not of consequence here, and the port id to 0, the kernel port/PID. We also sent the message via the genlmsg_multicast()</code> function. This is all for the kernel side of things for this tutorial. Now let's take a look at the user side of things. User land</h3> Netlink is a socket family and so it's possible to communicate over Netlink by just opening a socket and send and receiving messages over it, something like this: int fd = socket(AF_NETLINK, SOCK_RAW, NETLINK_GENERIC); /* Format request message... */ /* ... */ /* Send it */ send(fd, &req, sizeof(req), 0); /* Receive response */ recv(fd, &resp, BUF_SIZE, 0); /* Do something with response... */ /* ... */ </code></pre> That said, it's better to make use of the libnl^{1</a> library or similar since they provide a better way to interface with Generic Netlink that is less prone to errors and already contains all the boilerplate that you would need to write anyway. This library is precisely the one that I'll be using in this example.} We'll need to include some headers from the libnl library to get started: #include <netlink/socket.h> #include <netlink/netlink.h> #include <netlink/genl/ctrl.h> #include <netlink/genl/genl.h> #include <netlink/genl/family.h> </code></pre> As well as our shared header with the enumerations and defines from the kernel module: #include "../ks/genltest.h" </code></pre> I've also made a little helper macro for printing errors: #define prerr(...) fprintf(stderr, "error: " __VA_ARGS__) </code></pre> As I mentioned in the introduction, it's easier to use different sockets for unicast and multicast messages, so that's what I'm going to be doing here, opening two different sockets to connect to Generic Netlink: /* Allocate netlink socket and connect to generic netlink */ static int conn(struct nl_sock **sk) { *sk = nl_socket_alloc(); if (!sk) { return -ENOMEM; } return genl_connect(*sk); } /* * ... */ struct nl_sock *ucsk, *mcsk; /* * We use one socket to receive asynchronous "notifications" over * multicast group, and another for ops. We do this so that we don't mix * up responses from ops with notifications to make handling easier. */ if ((ret = conn(&ucsk)) || (ret = conn(&mcsk))) { prerr("failed to connect to generic netlink\n"); goto out; } </code></pre> Next we need to resolve the ID of the Generic Netlink family that we want to connect to: /* Resolve the genl family. One family for both unicast and multicast. */ int fam = genl_ctrl_resolve(ucsk, GENLTEST_GENL_NAME); if (fam < 0) { prerr("failed to resolve generic netlink family: %s\n", strerror(-fam)); goto out; } </code></pre> A (Generic) Netlink socket is not associated with a family, we are going to need the family ID when sending the message a little bit later. The libnl library can do sequence checking for us, but we don't need it for multicast messages, so we disable it for our multicast socket: nl_socket_disable_seq_check(mcsk); </code></pre> We also need to resolve the multicast group name. In this case we are going to be using the resolved ID right away to subscribe to the group and start receiving the notifications: /* Resolve the multicast group. */ int mcgrp = genl_ctrl_resolve_grp(mcsk, GENLTEST_GENL_NAME, GENLTEST_MC_GRP_NAME); if (mcgrp < 0) { prerr("failed to resolve generic netlink multicast group: %s\n", strerror(-mcgrp)); goto out; } /* Join the multicast group. */ if ((ret = nl_socket_add_membership(mcsk, mcgrp) < 0)) { prerr("failed to join multicast group: %s\n", strerror(-ret)); goto out; } </code></pre> We need to modify the default callback so that we can handle the incoming messages: /* Modify the callback for replies to handle all received messages */ static inline int set_cb(struct nl_sock *sk) { return -nl_socket_modify_cb(sk, NL_CB_VALID, NL_CB_CUSTOM, echo_reply_handler, NULL); } /* * ... */ if ((ret = set_cb(ucsk)) || (ret = set_cb(mcsk))) { prerr("failed to set callback: %s\n", strerror(-ret)); goto out; } </code></pre> As you can see, we set the handler function to the same for both sockets, since we will be basically receiving the same message format for both the do request and the notifications. Our handler looks like this: /* * Handler for all received messages from our Generic Netlink family, both * unicast and multicast. */ static int echo_reply_handler(struct nl_msg *msg, void *arg) { int err = 0; struct genlmsghdr *genlhdr = nlmsg_data(nlmsg_hdr(msg)); struct nlattr *tb[GENLTEST_A_MAX + 1]; /* Parse the attributes */ err = nla_parse(tb, GENLTEST_A_MAX, genlmsg_attrdata(genlhdr, 0), genlmsg_attrlen(genlhdr, 0), NULL); if (err) { prerr("unable to parse message: %s\n", strerror(-err)); return NL_SKIP; } /* Check that there's actually a payload */ if (!tb[GENLTEST_A_MSG]) { prerr("msg attribute missing from message\n"); return NL_SKIP; } /* Print it! */ printf("message received: %s\n", nla_get_string(tb[GENLTEST_A_MSG])); return NL_OK; } </code></pre> Nothing that fancy going on here, much of it is very similar to what we were doing on the kernel side of things. The main difference is that the message was already parsed for us by the kernel API, while here we have the option to walk the attributes manually or parse them all onto an array with the help of library function. Next we want to send a message to kernel space: /* Send (unicast) GENLTEST_CMD_ECHO request message */ static int send_echo_msg(struct nl_sock *sk, int fam) { int err = 0; struct nl_msg *msg = nlmsg_alloc(); if (!msg) { return -ENOMEM; } /* Put the genl header inside message buffer */ void *hdr = genlmsg_put(msg, NL_AUTO_PORT, NL_AUTO_SEQ, fam, 0, 0, GENLTEST_CMD_ECHO, GENLTEST_GENL_VERSION); if (!hdr) { return -EMSGSIZE; } /* Put the string inside the message. */ err = nla_put_string(msg, GENLTEST_A_MSG, "Hello from User Space, Netlink!"); if (err < 0) { return -err; } printf("message sent\n"); /* Send the message. */ err = nl_send_auto(sk, msg); err = err >= 0 ? 0 : err; nlmsg_free(msg); return err; } /* * ... */ /* Send unicast message and listen for response. */ if ((ret = send_echo_msg(ucsk, fam))) { prerr("failed to send message: %s\n", strerror(-ret)); } </code></pre> Also not that different from the kernel API. Now we listen once for the response to our command and indefinitely for incoming notifications: printf("listening for messages\n"); nl_recvmsgs_default(ucsk); /* Listen for "notifications". */ while (1) { nl_recvmsgs_default(mcsk); } </code></pre> As good hygiene, let's close the connection and socket before exiting our program: /* Disconnect and release socket */ static void disconn(struct nl_sock *sk) { nl_close(sk); nl_socket_free(sk); } /* * ... */ disconn(ucsk); disconn(mcsk); </code></pre> That's about it! Conclusion</h2> The old ways of interacting with the kernel and its different subsystems through such interfaces as sysfs and especially ioctl had many downsides and lacked some very needed features such as asynchronous operations and a properly structured format. Netlink, and its extended form, Generic Netlink, provide a very flexible way of communicating with the kernel, solving many of the downsides and problems of the interfaces of old. It's certainly not a perfect solution (not very Unixy for instance), but it's certainly the best way we have to communicate with the kernel for things more complicated than setting simple parameters. Post scriptum</h2> At the moment when I started learning to use Generic Netlink, the 6.0 kernel wasn't yet stable, hence the excellent kernel docs Netlink intro^{2</a> wasn't yet in the "latest" section, but rather in the "next" section. Since I was looking inside the then "latest" (v5.19) docs, I didn't notice it until I started writing my own article.} I'm not sure if I had started writing this article had I come across the new kernel docs page, but in the end I think it was worth it since this one is a good complement to the official docs if you want to get your hands dirty straight away, especially considering that it provides a practical example. Do give a read to the kernel docs page! It covers some things that might not be covered here. ^1Original site with documentation https://www.infradead.org/~tgr/libnl/</a>, up-to-date repository: https://github.com/thom311/libnl</a> </div> ^2Very good introductory Netlink article, from the kernel docs themselves: https://kernel.org/doc/html/latest/userspace-api/netlink/intro.html</a> </div>

Intro to Linux and the Bash command line, pt II

2018-12-31T09:20:00+00:00

So the year is coming to an end, and I didn't follow up on the first part of these mini tutorial/guide/introduction. Might as well end the year on a good note and write at least one post this last month.

In this part I will be writing about some more aspects of Linux and Unix-like systems and tools that will come in handy when using the terminal. All or most of these tools are a standard part of Linux and Unix/Unix-like systems.

File manipulation</h2>
I didn't get into detail in the previous part of this guide on the file manipulation side of things. However it is a crucial part of any computer system, after all they wouldn't be as useful as they are if you couldn't create, edit and delete files.

Create files</h3>

Usually, the way you would go about creating and editing files is by using the corresponding programs. Like for example, vim to create and edit text files. However, there's also a utility to create blank files called touch.

You can make a blank file by writing touch followed by the path of the file you wish to create, like this

user@host:~$ touch example.txt
</span></code></pre>
Now we have an empty file at "/home/user/example.txt". Although I kind of lied
when I said that this command is for making blank files. In reality, what it
does is check if such file exists, if it does, it modifies the date and time of
access and modification, otherwise it makes a new empty file, like in our case.
Most of the time you won't really need this command, since you would actually be
creating files through programs or means, though it does come in handy
sometimes.</p>
Create directories</h3>
Now on to something more interesting, making directories. This one is quite
simple, and the command for that is mkdir. For example</p>
user@host:~$ mkdir foo
</span></code></pre>
In this case we have created a new folder called "foo" in "/home/user/". But
remember that we are passing paths here, which can be relative or absolute (if
you don't know or remember what this means, check the first part of this guide).
In our case  we were using a relative path, with just the name of the directory
to create.</p>
If we were to indicate an absolute path like "/home/user/foo" we would need to
make sure that directories "home" and "user" already exist. However, there's a
useful argument we can pass to mkdir to make parent directories if needed, -p</p>
user@host:~$ mkdir -p /tmp/foo/bar
</span></code></pre>
In this case, if directory "foo" doesn't exist, mkdir will make it for us along
with "bar" inside it.</p>
Moving and renaming</h3>
This one is pretty simple, and both actions are handled by the same command, mv,
which is short for move.</p>
If you want to rename a file or directory, simply "move" it to the same
destination directory with a new/different name, for example</p>
user@host:~$ mv untitled titled
</span></code></pre>
If you want to move it to a different location, just indicate that path as the
second argument. Remember that we are using paths here, and we use either
absolute or relative paths</p>
user@host:~$ mv titled Documents/titled
</span></code></pre>
Copying</h3>
Copying files is similar to moving them, except that the command is different,
cp</p>
user@host:~$ cp titled Documents/titled2
</span></code></pre>
However, copying directories is different. To copy directories you have to use a
special flag, the -r flag. This flag means that the operation ought to be
recursive, that is to copy every file and subdirectory in that directory along
with the directory (and files in those subdirectories and files the
subdirectories of the subdirectories and... yeah recursion, you get it) to the
destination path. So we would do something like this</p>
user@host:~$ cp -r dir dir-copy
</span></code></pre>
Removing</h3>
Removing files is pretty simple, just use rm followed by the path of the file,
for example</p>
user@host:~$ rm title
</span></code></pre>
However, removing directories is a little bit trickier. One option would be to
use the command rmdir, however, the directory has to be empty before you can
remove it.</p>
The second option, is to use the same command as we used with files, but passing
it the -r flag so that it removes files recursively, similar to what we did with
the copy (cp) command. Do watch out, as it will remove the directory along with
everything that is in it.</p>
So to remove a directory and everything in it, we input something like this</p>
user@host:~$ rm -r dir
</span></code></pre>
This command also has another flag to forcefully remove files without prompting,
the -f flag. This might be useful when you are removing files recursively and
there might be some special files, like hidden files (the ones with a dot as the
first character) and you don't want to be prompted for each and everyone of
them.</p>
Thread REALLY carefully when using this command though, especially if you are
issuing it as root. You don't wanna issue a "sudo rm -rf /" and end up with a
borked system and lost files. Unless you are some kind of sadist or something.</p>
An example of when it might be useful, is when you need to delete a git
repository from your computer, since it contains a lot of special hidden files
which git uses to keep track of your commits and other stuff. So to remove it,
we do</p>
user@host:~$ rm -rf somerepo
</span></code></pre>
Permissions</h3>
The Unix or Unix-like user and permissions systems, is a robust system for
managing what particular users can do with certain files and directories. This
allows to create a secure environment, especially when there are multiple users
using one computer.</p>
Every file has three types of permissions which dictate what can be done with
it. Each permission is represented by a single letter</p>

r - read: the user can read/view the file.</li>
w - write: the user can write/modify (this includes deleting) the file.</li>
x - execute: the file can be run or executed by the user, if it is a program
or script.</li>
</ul>
There are as well three different sets of people the permissions and ownership
might apply to. These are</p>

u - user/owner: the one user that owns the file. Usually it is the user that
created the file, but ownership can be modified.</li>
g - group: every file has a group it belongs to. The group can be a one user
group (every user has their own one-user group), or a common group with
multiple users.</li>
o - others: everybody else who is not either in the group or the owner.</li>
</ul>
If you read the first part of this guide you might remember that I mentioned
using the command "ls -l" to list the contents of the directory with details
about the files, including permissions</p>
user@host:~/Documents$ ls -l
</span>drwxr-xr-x  2 user user  4.0K Jul  18 04:20 Books
</span>-rwxr-xr--  1 user group 350  Jul  18 04:20 run.py
</span>-rw-r--r--  1 user user  1.2M Jul  18 04:20 picture.png
</span></code></pre>
As mentioned in the first part, the first ten characters of a row of the output
is what tells us about the permissions of the given file in that row.</p>
The first character in that sequence tells us if it is a normal file, or a
directory. If it has a d, it is a directory (duh), if it has a dash, it is a
normal file. It can be another letter, like an l for a symbolic link (a topic
for another day).</p>
The next 9 characters are divided in subsequences of three letters, each for a
set of people. The first three characters after the file type one are the
permissions for the owner, the next three are for the group, and the last three
for others.</p>
In each of this subset of three characters, the permissions are in the order of
read, write, and execute. So a letter means that that said set of users have
said permission, and its absence means the lack of that permission.</p>
Let's take for example the "run.py" file. By the information there we can see
that it is a normal file (no d (that's what she said)), its owner (user) can
read, write and even execute it; the group users (in group group) can read and
execute it, but not write to it; and the others can just take a look at its
contents.</p>
You might have noticed that directories tend to have the "x" part of the
permissions, however, this means something a little bit different on directories
than on files. It doesn't mean that you can execute but rather that you can
"search it", or in other words, access files inside of it. Because having the
"read" permission on a directory only means that you can take a look at what
files are inside of it, and write that you can put files inside the directory.</p>
Changing permissions</h3>
There is a really useful command that allows us to change permissions on files,
chmod. It is short for change file mode bits.</p>
To change the permissions on the file you input chmod followed by the
permissions and the path of the file. There are two ways of setting the
permissions, the easy and long one using letters, and the short but not as easy
way with octal "permission bits". We'll take a look at the easy one first.</p>
The easier way is made up of three parts</p>

Who - user/owner, group, others or all (u, g, o, or a)</li>
Revoke or grant, "+" to grant, "-" to revoke</li>
The permission we are setting - read, write or execute (r, w, or x)</li>
</ul>
So let's suppose we have a script we want to set the execute permission for so
that any user in the computer can execute it.</p>
user@host:~/Documents/stuff$ ls -l
</span>-rw-r--r-- 1 user group  420  April 20 6:59  script.sh
</span>user@host:~/Documents/stuff$ chmod a+x script.sh
</span>-rwxr-xr-x 1 user group  420  April 20 6:59  script.sh
</span></code></pre>
As we can see, after executing the command, every user on the computer now has
execute permission on the script. Now let's say that we want to revoke read
permissions for everybody except for the owner (user), we could execute o-r and
then g-r, but we can also combine them, like so</p>
user@host:~/Documents/stuff$ chmod go-r script.sh
</span>-rwx--x--x 1 user group  420  April 20 6:59  script.sh
</span></code></pre>
Now onto the short way. This one's a bit harder to remember as you have to have
some understanding of binary. Basically the way it works is that you set the
permissions by passing three octal numbers (i.e., 0-7) that each represent the
permission bits for each set of people (user/owner, group, others).</p>
As it is we have three possible permissions (read, write and execute) and 2^3
just so happens to be 8 (possible combinations), that's why it is in octal.
Here's a table to help you out</p>

  
    
      Octal</td>
      Binary</td>
    </tr>
    

      0</td>
      001</td>
    </tr>
    

      1</td>
      001</td>
    </tr>
    

      2</td>
      010</td>
    </tr>
    

      3</td>
      011</td>
    </tr>
    

      4</td>
      100</td>
    </tr>
    

      5</td>
      101</td>
    </tr>
    

      6</td>
      110</td>
    </tr>
    

      7</td>
      111</td>
    </tr>
  </tbody>
</table>
Basically what this means, is that when we have a 1, said permission is granted,
when it is 0, it is not. So for example, 6 means that read and write have been
granted, but not execute, because 6 => 110 => rw-.</p>
So let's say for example that we want to set the permissions of file so that the
owner can read, write, and execute, the group can read and execute, and others
can only execute. It would look something like this</p>
user@host:~/Documents/stuff$ chmod 751 script.sh
</span>user@host:~/Documents/stuff$ ls -l
</span>-rwxr-x--x 1 user group  420  April 20 6:59  script.sh
</span></code></pre>
If you are changing the permissions of a directory, the permissions will apply
only to the directory itself and not to the files and subdirectories inside of
it, unless you use the recursive flag -R (note that in the case of chmod and
chown it is a capital R).</p>
Changing ownership</h3>
Changing the owner of a file is easier than changing permissions, since less
variables are involved. To change owner we use the chown command, for example,
change the user that owns a file</p>
user@host:~/.logs$ ls -l
</span>-rw-r--r-- 1 root root  69  April 20 6:59  some.log
</span>user@host:~/.logs$ sudo chown user some.log
</span>user@host:~/.logs$ ls -l
</span>-rw-r--r-- 1 user root  69  April 20 6:59  some.log
</span></code></pre>
As the file doesn't belong to the same user initially, we need to use sudo
elevate our permissions to "super user". Note that in order to be able to use
sudo or "elevate your permissions" your user needs to be either in the sudoers
file or in other distros in the "wheel" group, or both. I won't go into details
on how to do that, since most probably your user is already in the sudoers file,
and a quick search on the internet will give you the information needed.</p>
Now let's say that you wanted to change both the user and group that file
belongs to, we would do like this</p>
user@host:~/.logs$ ls -l
</span>-rw-r--r-- 1 root root  69  April 20 6:59  some.log
</span>user@host:~/.logs$ sudo chown user:group some.log
</span>user@host:~/.logs$ ls -l
</span>-rw-r--r-- 1 user group  69  April 20 6:59  some.log
</span></code></pre>
Just as with chmod, if you want to change ownership recursively in a directory,
you need to use the -R flag.</p>
Epilogue/Some tips</h2>
I forgot to mention in the previous part how to write paths with spaces. You
cannot write paths with spaces just like that, since spaces are treated as the
separator between arguments.</p>
There are two ways of writings paths and filenames (and other arguments) with
spaces. One is to put the argument between quotation marks, like for example</p>
user@host:~/Documents$ rm "Shopping List.txt"
</span></code></pre>
Another is to use an escape character, i.e. just put a backslash () before the
space, like this</p>
user@host:~/Documents$ rm Shopping\ List.txt
</span></code></pre>
One more thing that I wanted to mention is the man(ual) pages. Basically all
Linux distros come with man pages.</p>
You don't always have to resort to the internet if you don't remember how to use
a command, like what flags a command accepts, what are the order of the
arguments, etc. You might not have internet one day, and you might to resort to
good ol' offline resources, and the man pages are a good one.</p>
Besides, it is sometimes faster and easier to not have to leave the terminal to
look up the information.</p>
To read the man(ual) pages of a command, input "man" followed by the name of the
command, e.g. "chmod", and it will provide you with almost all the information
that you need.</p>
You can use vim keys (hjkl), or the arrow keys to scroll. Type / to search
for that word, and afterwards press n to find the next found item.</p>
This is all for this part. Happy 2019!</p>


Intro to Linux and the Bash command line
2018-11-05T23:53:00+00:00
Recently I decided to introduce a friend of mine to the wonderful world of
Linux, and like when someone moves to a completely new town, you have to help
that friend get around town and learn about how things work in this new town,
where are all the places of interest, etc. And so it is, in someway, when
someone decides to make the move to a new OS, they have to get used to the new
environment and make new habits, especially regarding Linux. Therefore, I
decided to write this tutorial for my friend, and anybody who decided to try
Linux, and want to learn to use more effectively, that is, with a higher level
of skill than an average user.</p>
</span>
Most probably you already know, but Linux is not actually the OS itself, but
rather the kernel that is used in conjuction with a collection of programs and
packages called GNU (insert here jokes and memes about GNU/Linux (or
GNU+Linux)), and there are many different distributions (over 9000) of said
GNU/Linux.</p>
If you're just about to install Linux, and haven't decided yet on a distro, here
I wrote a small list of distros that might start with:</p>

Manjaro - I personally use this distro. This one has different releases, with
different Desktop Environments (or DEs), like KDE or GNOME. This is a "rolling
release", what that means, is that instead of the classical update to version
x.x of the OS system, updates for each package are being rolled out
constantly, and sepparatly for each package. That is, you get basically the
latest packages, with the latest updates, at the cost of maybe some stability.
It is quite easy to get installed, and get started with (If you choose an
official release like Manjaro KDE or Manjaro GNOME).</li>
Ubuntu - The most know Linux distribution. It is one of the most
"user-friendly" distros out there. It is more stable than Manjaro (although
not always, unless you're using the Long-Term Support versions). If you just
want to install an OS where everything just works, and is already configured
for you, and you don't want to choose from a list of DEs (or you don't know
what a Desktop Environment is), this will most probably suit your needs.</li>
Debian - One the most stable distributions. It is not as easy to install as
Ubuntu or Manjaro due to large ammount of choice given during installation,
but it's not really hard to either. Actually, Ubuntu is derived from Debian.
You must probably will have to configure it to your needs and likings
(personally, I dislike the default settings of GNOME), but if you want real
stability, and distro that has been around almost as long as the kernel
itself, then this might be the distro for you. One downside of Debian, is that
because of its focus on stability, many packages on the main (stable)
branch/version are quite old/outdated. Another con of Debian, is that it
doesn't include "non-free" packages in the main repository (that is, there are
no proprietary packages by default), so if need a proprietary program or
package, like nvidia's drivers, you will have to add the corresponding
repositories by yourself.</li>
Fedora - It is similar to Ubuntu regarding stability and ease of use (i.e.
quite simple to use). I can't say much about Fedora, because I haven't used it
that much myself, but it is a very popular distribution. It has the same con
as Debian, in that it doesn't include "non-free" software in its main
repository.</li>
</ul>
If you found "Desktop Environment" to be a unfamiliar term, in a nutshell, it is
the collection of programs and packages that present with the Graphical User
Interface for interaction with the system. How your OS/distribution is going to
look like doesn't depend as much on the distro itself, as on the DE that you
choose to install or comes with the distro.</p>
Do you have Linux installed? Excellent, now we can get started.</p>
File structure</h2>
If you are used to working with Windows systems, the the first thing that you
might notice is the difference in how files are handled/organized. In Windows,
because of its DOS legacy, drive letters are used to represent different drives,
partitions and file systems. In Linux, like in other Unix-like systems (e.g.
macOS, BSD) this differs somewhat.</p>
In Linux, everything is a file, including the devices that are connected to your
computer. From your keyboard (which is a read-only file) to your drives.
Directories are also files.</p>
Different disks or drives are mounted in a specific directory, and from this
directory, every file from that drive will be accessible. You can mount and
unmount drives yourself, but if you installed a distro with any of the most
popular desktop environments, then there's nothing to worry about, the system is
going to take care of mounting your pendrive for you, and creating a shortcut in
your file explorer and/or desktop each time you insert it.</p>
Linux has what is called a "root" directory</p>
/
</span></code></pre>
In there are the files and subdirectories in your system are located, including
drives mounted in a specific subdirectory as mentioned before.</p>
Each user in Linux has its own "home" directory. Inside your home directory your
personal documents/files and subdirectories are going to be stored. All the home
directories of each user are usually going to be stored inside the "/home/"
directory. For example, for user "user", their home directory is going to be</p>
/home/user
</span></code></pre>
However, you can also move to your home directory by using the symbol ~. For
example, in your terminal, if you input</p>
$</span> cd </span>~
</span></code></pre>
You are going to be taken to your home directory (e.g. "/home/user/").</p>
The command-line - Bash</h2>
Arguably the most useful program in Linux and any *nix system is the terminal.
Yes, maybe the average user won't have to use it, but it is the most flexible,
effective and useful instrument in your computer. A lot of work can be done
faster and more effectively in the terminal, rather than in a GUI. Obviously, it
is faster to learn to use a graphical interface, than a text-based or
command-line one, however, once you learn to properly use the terminal, you will
be able to use computer more efficiently than ever.</p>
Your command-line, if you haven't changed any defaults, will most probably look
something like this</p>
user@host:~$
</span></code></pre>
The first part before the "@" symbol, is the user with whom you have logged in.
After the "@" symbol is the "hostname" of your machine, i.e. the name of your
computer in a network.</p>
After the semicolon ":", the directory in which you are currently located is
going to be displayed. In our case, the "~" symbol is displayed, meaning that we
are inside our home directory. If you wish to see the full absolute path in
which you are located, you can input the command "pwd"</p>
user@host:~$</span> pwd
</span>/home/user
</span>user@host:~$
</span></code></pre>
The dollar "$" symbol, tells us that we are logged in as "normal" user. In Linux
and all *nix systems, there is a so called "super user", or just the "root" user
for short. Normal users don't have full access to all of the files in the
system, including other users' files. The root user has full access to all
system files. When you are logged in as the root user, the dollar "$" symbol is
going to be replaced by the hash "#" symbol.</p>
However, we will be talking about the root user and permissions another time.</p>
I can't tell you exactly how to open the terminal, as it is different in each
distro and desktop environment. You will have to look for the link that says
"Terminal" in your programs' menu, with the icon of a terminal on it.</p>
To be able to start working in the terminal, we'll need to become acquainted
with some of the main commands used for navigating.</p>
Navigation in the terminal</h3>
There is one more thing that needs to be taken into consideration when working
with Linux and Unix-like systems, and that is, case-sensitivity. *nix systems
are case-sensitive, in contrast to Windows ones, which aren't. What that means,
is that, if on Windows, files "README.TXT" and "readme.txt" are the same file,
in Linux and Unix they are completely different files.</p>
To move around directories, we use the "cd" (or change directory) command,
followed by the name, or path of the directory. The directory can be absolute or
relative.</p>
A relative path, is, for example, the name of a subdirectory inside the
directory in which we are currently located. In other words, the path relative
to where we are located.</p>
An absolute path, is the path relative to the root directory.</p>
For example, if we want to move to the "Documents" directory inside
"/home/user/", and we are already at "/home/user/", the we can just type the
following</p>
user@host:~$</span> cd Documents
</span>user@host:~/Documents$
</span></code></pre>
If we wanted to move to that same directory from another location in the system,
we would type</p>
user@host:/var$</span> cd /home/user/Documents
</span>user@host:~/Documents$
</span></code></pre>
To move one folder up, we write "..". Example</p>
user@host:~/Documents$</span> cd ..
</span>user@host:~$
</span></code></pre>
The two dots "..", means the root directory of the current subdirectory. One dot
"." means the current directory.</p>
There is one more command, that will aid you in navigating - ls. This command
outputs a list of files and directories in the current location</p>
user@host:~/Documents$</span> ls
</span>Books</span>       todo.txt    picture.png
</span></code></pre>
This command also accepts arguments and flags. For example, to list hidden files
as well, add the "-a" flag</p>
user@host:~/Documents$</span> ls</span> -a
</span>Books</span>    .secret    todo.txt    picture.png
</span></code></pre>
Hidden files in Linux start with a dot (e.g., hidden directory ".secret").</p>
There is also the "-l" parameter, which shows us a list of the contents with
additional information, such as, permissions (more on that in the next part),
amount of files inside a directory, owner of the file (user and group), size on
disk, datetime of creation/modification, and the name of the file. Example</p>
user@host:~/Documents$</span> ls</span> -l
</span>drwxr-xr-x</span>  2 user user 4.0K Jul  18 04:20 Books
</span>-rw-r--r--</span>  1 user user 350  Jul  18 04:20 todo.txt
</span>-rw-r--r--</span>  1 user user 1.2M Jul  18 04:20 picture.png
</span></code></pre>
You can mix flags as well,</p>
user@host:~/Documents$</span> ls</span> -al
</span>drwxr-xr-x</span>  2 user user 4.0K Jul  18 04:20 Books
</span>drwxr-xr-x</span>  5 user user 4.0K Jul  18 04:20 .secret
</span>-rw-r--r--</span>  1 user user 350  Jul  18 04:20 todo.txt
</span>-rw-r--r--</span>  1 user user 1.2M Jul  18 04:20 picture.png
</span></code></pre>
You can also take a look at what is inside a directory without having to move to
it first, by passing along as the last argument the name/path of said directory,
for example</p>
user@host:~/Documents$</span> ls</span> -l</span> Books
</span>drwxr-xr-x</span> 12 user user 4.0K Jul  18 04:20 Lessons
</span>-rw-r--r--</span>  1 user user 2.3M Jul  18 04:20 Crime and Punishment.pdf
</span></code></pre>
Shortcuts</h3>
Before we conclude the first part of this tutorial, I would like to mention some
useful "shortcuts" in bash.</p>
The first one is command history. Each time you input a command into the
terminal, it saves it in a history file. You can move around your command
history by using the up and down arrows. Let's say you want to repeat the last
command you used, instead of typing it all over again, you could just hit the up
arrow one time, maybe modify it a bit, and then press enter to input it. If you
want to use an older command, you can press the up arrow multiple times, and if
you missed the command you needed, you can go forward in your history by
pressing the down arrow.</p>
There is one more useful thing in Bash - Tab completion. When you press the
"Tab" key, bash will try to autocomplete the command for you.</p>
Let's say, for example, that you are in the root directory and you want to move
to "/home/user/". you can start typing "cd h", then press "Tab", so you now have
"cd home/", now type "u", press "Tab" one more time, and now you have "cd
home/user/".</p>
If there are multiple possible options to be autocompleted, then on the first
hit of the "Tab" key you are not going to get anything. If you don't get
anything it could also mean that there are nothing to autocomplete. In the first
case, if you press "Tab" twice, bash is going to output a list of possibilities,
for example</p>
user@host:~$</span> cd Do
</span>Documents/</span> Downloads/
</span>user@host:~$</span> cd Do
</span></code></pre>
In this case, we can type the next letter, for example "cd Doc", and press "Tab"
one more time, and we are going to get "cd Documents/". However, if you didn't
get anything on pressing "Tab" twice, it means that is nothing to autocomplete.</p>
It can be a little be hard to understand at first how "Tab completion" works,
but the best way to understand it, by trying it yourself.</p>
That is all for now, in the next part I am going to talk more about working on
the terminal.</p>


Setting up your own VPN
2018-11-02T05:43:00+00:00
There are many reasons why you would want to use a VPN, especially in this day
and age. The major one would be to securely and more privately surf the web, but
that is far from the only reason. I for one use it as well to be able to set up
a remote network with my desktop PC while I'm away from home, so that I can
access all of my files through SSH. There are a ton of other reasons as to why a
VPN could be useful, but that is not the topic of this post, rather I will be
writing here about to set up your own VPN (yes your own!) on a VPS, or any other
kind of linux server for that matter, using OpenVPN.</p>
</span>
I am in fact running my own VPN in the same server that I use to host this site
and other projects of mine. However due to certain circumstances I need to
migrate to another VPS, and so I decided to write this little guide to refresh
my memory on how to set up OpenVPN.</p>
Before we begin to set up our VPN, we need to install OpenVPN. I am using Debian
Stretch, so your install process may differ depending on your distro.</p>
# apt-get install openvpn easy-rsa
</span></code></pre>
Configuring OpenVPN</h2>
After installing the necessary packages, we proceed to configure OpenVPN. First
we need to extract the sample OpenVPN server configuration file to /etc/openvpn</p>
# gunzip -c /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz > /etc/openvpn/server.conf
</span></code></pre>
Next we proceed to edit it.</p>
# vim /etc/openvpn/server.conf
</span></code></pre>
Inside we will make changes to basically use higher-level encryption, forward
web traffic to destination, prevent DNS leaking, and set up permissions.</p>
First we make sure that we are using 2048 bit-length keys instead of 1024. In my
case it was already set to 2048 bit by default, however, the first time that I
set up OpenVPN the default was 1024, so find this line</p>
dh dh2048.pem
</span></code></pre>
If instead you see "dh1024.pem" change it so that it is like in this example.</p>
Next we make sure to redirect all traffic to its proper destination. Find this
line</p>
push "redirect-gateway def1 bypass-dhcp"
</span></code></pre>
If it has a semicolon (;) at the beginning, remove it so that it is uncommented.</p>
Now we will tell OpenVPN to use OpenDNS for DNS resolution. This will help
prevent DNS requests from leaking outside of the VPN connection.</p>
Immediately after the previous setting, you will see a block of comments
followed by two commented lines</p>
push "dhcp-option DNS 208.67.222.222"
</span>push "dhcp-option DNS 208.67.220.220"
</span></code></pre>
As previously, just remove the semicolon character from the beginning of the
line.</p>
Next we should comment the next line with a # or ; character. It is supposed to
be used as an extra security measure especially against DoS attacks, however, we
don't really need it</p>
tls-auth ta.key 0 # This file is secret
</span></code></pre>
If you really want to leave that on, you will need to generate it</p>
openvpn</span> --genkey --secret</span> ta.key
</span></code></pre>
And last but not least in our server.conf file, we need to adjust permissions.
Find and uncomment these lines</p>
user nobody
</span>group nogroup
</span></code></pre>
This way OpenVPN doesn't run as the root user. And we don't want any random
program wildly running as root, especially ones that are exposed to the
interwebz, no sir we don't ( ͡° ͜ʖ ͡°).</p>
After having made all the needed changes, just write them and close the editor.</p>
Configuring network and firewall</h2>
We need to enable packet forwarding, otherwise our traffic will not leave the
server.</p>
So that we don't have to reboot our server (let's leave all that rebooting to
windows systems), we can enable it on runtime by running the following command</p>
# sysctl net.ipv4.ip_forward=1
</span></code></pre>
However, we need to make this change permanent as well, so we open the following
file</p>
# vim /etc/sysctl.conf
</span></code></pre>
And uncomment the following line by removing the hash character (#) at the
beginning of the line</p>
net.ipv4.ip_forward</span>=</span>1
</span></code></pre>
Save your changes and close the editor.</p>
Now we need to properly set up our firewall. Personally I use ufw as front-end
to iptables, since it is pretty easy to set up and use (as its name suggests,
uncomplicated firewall). If you don't have it installed yet, you can go ahead
and run</p>
# apt-get install ufw
</span></code></pre>
By default ufw denies all incoming connections and allow all outgoing. Those
defaults are fine for my case, some might prefer to deny all outgoing
connections by default, some riskier guys (or gals) might prefer to allow all by
default (however, you wouldn't have unprotected fun time with just any partner,
now would you?( ͡° ͜ʖ ͡°)), however explaining all the details about ufw is out of
the scope of this tutorial, but I invite you use your duckduckgo-fu if you're
interested in learning more about ufw.</p>
Now, as ufw denies all incoming connections by default, we need to configure it
so that we don't get locked out of our server. If you use the standard ssh port,
just run the following command</p>
# ufw allow ssh
</span></code></pre>
ufw comes with some presets for the most common services, so that in fact allows
connections from port 22 over tcp. If you, like me, prefer to use another port
for SSH, you need to specify the port and protocol manually, so, if say, you
connect to SSH over port 3333, you would run</p>
# ufw allow 3333/tcp
</span></code></pre>
(Of course that is not the actual port I myself use for ssh, so don't even try
(That is however not an open invitation to try and guess my SSH port and hack
yourself into my server (seriously, please don't hack me ´༎ຶ ͜ʖ ༎ຶ ))).</p>
In this tutorial we will be using OpenVPN on port 1194 over UDP, so we must
allow it</p>
# ufw allow 1194/udp
</span></code></pre>
Next we need to set ufw's forwarding policy, so we open the primary
configuration file</p>
# vim /etc/default/ufw
</span></code></pre>
Modify the following line</p>
DEFAULT_FORWARD_POLICY="DROP"
</span></code></pre>
So that it looks like this</p>
DEFAULT_FORWARD_POLICY="ACCEPT"
</span></code></pre>
Save and exit.</p>
Open this file</p>
# vim /etc/ufw/before.rules
</span></code></pre>
And add the rules for OpenVPN somewhere after the first block of comments</p>
# START OPENVPN RULES
</span># NAT table rules
</span>*nat
</span>:POSTROUTING ACCEPT [0:0]
</span># Allow traffic from OpenVPN client to eth0
</span>-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
</span>COMMIT
</span># END OPENVPN RULES
</span></code></pre>
Once more, save and exit.</p>
Now we can safely enable ufw</p>
# ufw enable
</span></code></pre>
It will say something about disrupting current SSH connections, however, since
we just added the needed rules for SSH we can go ahead and answer y.</p>
Configuring the Certificate Authority</h2>
Now we are going to setup our own CA. This is crucial since OpenVPN encrypts
traffic (what use is a VPN that doesn't encrypt traffic?)</p>
First we need to copy the RSA generation scripts</p>
# cp -r /usr/share/easy-rsa/ /etc/openvpn
</span></code></pre>
Next, we create a directory to house our keys</p>
# mkdir /etc/openvpn/easy-rsa/keys
</span></code></pre>
Now we open the variables files</p>
# vim /etc/openvpn/easy-rsa/vars
</span></code></pre>
And modify these lines according to our needs/preferences (they are not that
really important)</p>
# These are the default values for fields
</span># which will be placed in the certificate.
</span># Don't leave any of these fields blank.
</span>export </span>KEY_COUNTRY</span>=</span>"</span>US</span>"
</span>export </span>KEY_PROVINCE</span>=</span>"</span>CA</span>"
</span>export </span>KEY_CITY</span>=</span>"</span>SanFrancisco</span>"
</span>export </span>KEY_ORG</span>=</span>"</span>Fort-Funston</span>"
</span>export </span>KEY_EMAIL</span>=</span>"</span>me@myhost.mydomain</span>"
</span>export </span>KEY_OU</span>=</span>"</span>MyOrganizationalUnit</span>"
</span></code></pre>
Then, in the same file, we need to change the name of the key. For simplicity's
sake we will use the name "server", since that's the name that OpenVPN uses to
reference the .key and .crt files by default. If you decide to use a different
name, you will need to modify the OpenVPN configuration files that reference the
aformentioned files.</p>
So change this</p>
# X509 Subject Field
</span>export </span>KEY_NAME</span>=</span>"</span>EasyRSA</span>"
</span></code></pre>
Into this</p>
# X509 Subject Field
</span>export </span>KEY_NAME</span>=</span>"</span>server</span>"
</span></code></pre>
Save and exit.</p>
Next we will use OpenSSL to generate the Diffie-Helman parameters. Grab a cup of
tea, take a seat, it might take some minutes</p>
# openssl dhparam -out /etc/openvpn/dh2048.pem 2048
</span></code></pre>
Now that our certificate is ready, it's time to generate a key. For that, we
first switch into the easy-rsa folder</p>
# cd /etc/openvpn/easy-rsa
</span></code></pre>
Now we start setting up the CA. We first, need to initialize the Public Key
Infrastructure (PKI). For that we need to source the vars file</p>
# . ./var
</span></code></pre>
If you get something like this in your output</p>
  No /etc/openvpn/easy-rsa/openssl.cnf file could be found
</span>  Further invocations will fail
</span></code></pre>
Your .cnf might have a different name (due to differing versions of OpenSSL).
Just copy it like this</p>
# cp openssl-*.cnf openssl.cnf
</span></code></pre>
And source the vars file one more time.</p>
After sourcing the vars file, it will give us a warning about removing some
files. Don't sweat it, it's fine, since right now that folder is empty. So we go
ahead and clear any and all keys that might interfere with out setup</p>
# ./clean-all
</span></code></pre>
Finally, we can go ahead and build our CA using the following command</p>
# ./build-ca
</span></code></pre>
We can skip through each prompt since we set up that info previously inside the
vars file. The CA is now ready to go.</p>
Generating a certificate and key for our server</h2>
Now we can proceed to actually setting up and launching OpenVPN.</p>
While still on the easy-rsa folder, we input this command</p>
# ./build-key-server server
</span></code></pre>
server is the name of the key that we set up earlier in the vars file. We can go
ahead, and as earlier leave the defaults which we configured in the vars file.</p>
Upon reaching these prompts</p>
A challenge password []:
</span>An optional company name []:
</span></code></pre>
Just press enter for both. We won't be needing them for this setup.</p>
The last two entries require a y answer</p>
Sign the certificate? [y/n]
</span>1 out of 1 certificate requests certified, commit? [y/n]
</span></code></pre>
Now that our server certificate and key are ready, we need to copy them over to
/etc/openvpn, since that the directory in which OpenVPN will be looking for them</p>
# cp /etc/openvpn/easy-rsa/keys/{server.crt,server.key,ca.crt} /etc/openvpn
</span></code></pre>
Now we can start OpenVPN</p>
# systemctl start openvpn
</span></code></pre>
You can check if it launched correctly by running</p>
# systemctl status openvpn
</span></code></pre>
It should say something like "Active: active (exited) since..." then,
congratulations! OpenVPN is now running in your server. If it says otherwise,
maybe something like "inactive (dead)...", you might to check the log file
"/var/log/syslog" for errors.</p>
If all is fine and dandy, you can enable the OpenVPN service so that it
automatically starts on each reboot</p>
# systemctl enable openvpn
</span></code></pre>
Generating certificates keys for clients</h2>
Ideally each client that needs to connect to the VPN should have their own
unique certificate and key. Actually, by default, OpenVPN doesn't allow
simultaneous connections using the same certificate.</p>
Because of that you will need to repeat the steps in this section for each
device that you wish to allow to connect to your VPN, just changing the name
"desktop" below, to something different like "laptop" or "phone". This way you
can also later deactivate a specific device/client from accessing if needed be.</p>
I will start by building the key and certificate for my desktop computer, so I
will be using the "desktop" name as an example. We still should be in our
easy-rsa directory</p>
# ./build-key desktop
</span></code></pre>
Once more, as with the server's cert and key, we will asked about some info
which we set on the vars, so we should skip all that by pressing enter
(including setting the passwords). Just press y both times at the end to
confirm.</p>
Now we need the sample profile to the keys folder. We need to change its
extension to ovpn, since that will be the file that OpenVPN will use on each
device to automatically set the profile to connect to our VPN server</p>
# cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf /etc/openvpn/easy-rsa/keys/client.ovpn
</span></code></pre>
The name of this file doesn't need to be related to our client's cert and key
filenames, we can actually name to something that will describe our VPN, since
that will be the name that OpenVPN will be pick up by default in our client
device for our profile. So you can name it "work.ovpn" or maybe something more
rich like "muhfreedumbs.ovpn"</p>
# mv keys/client.ovpn keys/muhfreedumbs.ovpn
</span></code></pre>
After that we need to make some modifications to our ovpn file, so open it up</p>
# vim keys/muhfreedumbs.ovpn
</span></code></pre>
And edit the line starting with remote with your server's ip address</p>
remote your.ip.address.here 1194
</span></code></pre>
And if you chose to leave out the "ta.key" file protection option, don't forget
to comment this line</p>
tls-auth ta.key 0
</span></code></pre>
Next we find these two lines and uncomment them (remove the ; character at the
beginning of each line)</p>
user nobody
</span>group nogroup
</span></code></pre>
That's it. All that's left is to copy the client key and certificate, and the
ovpn file over to our device. First we need to copy them to another folder that
is available to our standard (non-root) user on the server for reading, since it
is not a good idea to have root login enabled on SSH. For example, we could
create a "vpnkeys" folder in our user's home directory</p>
$</span> mkdir /home/yaroslav/vpnkeys
</span></code></pre>
And then copy the need files over to that folder</p>
# cp /etc/openvpn/easy-rsa/keys/{desktop.key,desktop.crt,muhfreedumbs.ovpn} /home/yaroslav/vpnkeys/
</span></code></pre>
There is one more file that we need to copy, the server's certificate</p>
# cp /etc/openvpn/ca.crt /home/yaroslav/vpnkeys/
</span></code></pre>
If you previously also left the secret "ta.key" file enabled and generated it,
you should copy that too.</p>
Note that the "ca.crt" and the ovpn files will be the same for all devices. If
you wish to create a new key/cert pair for another device/client, you DO NOT
need to create a new ovpn file, just run the "./build-key clientkeyname" command
as before and copy the same old ovpn and ca.crt file along with client key and
cert to your other device.</p>
Now we need to actually copy the keys to our device. You can use rsync or scp
for that. For example, I will be using scp to copy them to my Keys folder</p>
$</span> scp</span> -P</span> portno</span> -r</span> yaroslav@ipaddressorurl:/home/yaroslav/vpnkeys </span>~</span>/Keys/
</span></code></pre>
If you need to use your VPN on your mobile device, just copy them first to your
computer through SSH, and then transfer them over to your mobile device.</p>
After transferring them to the device, you should delete the keys from the home
directory.</p>
Conclusion</h2>
Now that all is setup on the server side of things, you can go ahead and add
your VPN profile on your device. This process will vary greatly from device to
device.</p>
On most Linux distros/DEs, for example, you would go to your network manager,
and select "Configure VPN..." or something like that. Then, a window with a list
of (VPN) connections will appear, and you should be able to add your profile by
clicking on the plus (+) button. After another window like this should pop up</p>
</p>
You should select the option that says "Import a saved VPN configuration..."</p>
Note that the "openvpn" and "networkmanager-openvpn" packages should be
installed in your (client) system.</p>
Once you have connected to your VPN, you can head over to
https://dnsleaktest.com/</a> to check your connectivity.</p>
Now you can more safely and securely browse the interwebz, and be sure that no
third-party VPN company is logging your activity.</p>


Some thoughts on web design
2018-10-20T02:25:00+00:00
I am no designer by any means, I mostly prefer to deal with the technical side
of things. However I do appreciate when things aren't an eyesore, and most
importantly regarding technology, when the design doesn't get in the way. That
is, I value more UX, than an overtly designed UI.</p>
</span>
Early on websites were designed to just contain information, things like text
and images, without any of the fancy things that come with today's websites.
Nowadays it seems almost like a must that a website should have some kind of
interactivity, even if it doesn't add any actual value to the website.</p>
I am not trying to say that websites should stay the same static things that
they were in the past, rather, that a lot of bloat has been added to them in an
attempt to make them more "appealing" and interactive, which is not a bad thing
in and of itself.</p>
However, that trend has spawned a lot of bloated websites with unnecessary or
even annoying gimmicks. Gimmicks like menus that open up automatically blocking
the view of what you are reading, and assume that you actually need it at the
moment. Or the worst offender, autoplaying videos.</p>
Setting aside my rants about the modern offenses of design and UX in web
development, I want to focus on a couple of issues.</p>
Typography/Readability</h2>
I have started to notice lately that a great deal of
websites tend to neglect what I think is one of the most important aspects in a
good user experience (maybe except for media sites like youtube), and that is
the text itself.</p>
The people who design these sites seem to pay too much attention to the overall
style of the site and its elements, but don't seem to care about the actual
legibility of them. One might think that font type and size should be enough,
but those are far from the only important aspects to consider for comfortable
reading. Line length and spacing are also key aspects, and might be even more
important than the font you choose (just don't choose some quirky font like
Comic Sans, for your readers sanity's sake).</p>
I came to the realization that line length is a really important aspect for a
comfortable read, after having worked some time with LaTeX and wondering why it
always defaulted to the same line length. I always noticed that documents
produced in LaTeX were more pleasant to read than, for example, Word documents,
and it was not only the font.</p>
So after some reading on the internet about the choices of the default styles
for LaTeX, I realized how important line length is. I now understood why even
PDFs were more pleasant to read than your average website.</p>
Consider a book. Have you ever read a book whose line length was more than 80
characters? If so, that's a terrible book.</p>
There seems to be some consensus that an optimal line length should be between
50-75 characters, spaces included (right now my blog's articles are at about 80
characters, so there's room for improvement).</p>
If the line is too narrow, it breaks the reader's rhythm, and stresses them. If
it is too wide, the read will have a harder time focusing on the text, it will
be harder to gauge where the line ends and the next begin, and the reader's eyes
will end up moving more horizontally, also stressing the reader in the end.</p>
A lot of websites tend to present the text in lines too long for comfortable
reading. That factor ends up contributing in more readers skimming through text
and not paying much attention to what has been written.</p>
One more thing would be line height. I've noticed that many websites tend to use
most of the default styles for text, and sometimes the line height ends up being
too small for the font used and the lines of text being crammed. It's not as bad
as with the line length, but it is a problem present in some websites.</p>
Bloat</h2>
Oh boy, there's a lot that can be said about bloat on modern websites.
I'll try to keep it short. I'll begin by writing about some "addons" that end up
making sites heavier than they should.</p>
I feel like for most use cases JQuery is a really unnecessary thing to use. Most
website don't need half the features that JQuery provides, and most of its
features are just "shortcuts" that JS already offers by default. I mean, if
you're using JQuery, most of the time, you're being lazy. JQuery makes your site
unnecessarily heavier, although if your site is already JS-heavy, and the focus
of your site is actually on providing some specific functionality, JQuery can be
a great tool.</p>
Another thing that bloats sites, are CSS frameworks (e.g. Bootstrap). They can
make life easier when building templates for your sites, but most of the time,
as is the case with JQuery, you're not going to use half the features it
provides. In this case, it can get worse, since you will most probably end up
overriding a great deal of the default style provided with the framework anyway,
especially if you don't want your site to look generic.</p>
Finally, on the topic of bloat, is feature and visual bloat.</p>
The biggest offenders I have found to be news sites, especially those kinds of
"specialized" magazine type of sites. You enter the site to read the article,
and most of your screen is occupied by ads, useless images and links to other
articles and associated websites that distract you from your reading.</p>
And don't get me started on the autoplaying videos (I'm looking at you CNN,
although I don't even know why I bother to read their biased bullshit</del>
articles).  If I opened up an article, I want to read. READ, not watch or listen
some dude babbling about what I'm intending on reading. If I wanted to watch a
video, I would have gone to the videos section of the site, or to YouTube.</p>
Might be just my humble opinion, but when I am reading, I like to see only the
text in front of me, with maybe some links to the sides or top/bottom, to help
me navigate, and maybe some images relevant to what I am reading.</p>
Conclusion/Final words</h2>
Keep those lines at a sane width, let your lines and characters breathe with a
little bit of space, and maybe up your font size a bit if you notice you start
squinting when trying to read.</p>
Fluid design doesn't mean that the container where your text is should occupy
the entire width of your screen.</p>
Make beautiful, but lightweight (in bytes, and visually) websites, that are
comfortable to read, and your readers will thank you.</p>
Some links that might be of interest:</h3>

https://baymard.com/blog/line-length-readability</a></li>
https://www.paulolyslager.com/optimal-text-layout-line-length/</a></li>
http://bettermotherfuckingwebsite.com/</a></li>
</ul>


New w3blog release, version 0.5
2018-10-10T12:04:00+00:00
After some months without much free time, I finally could sit down a week ago
and start working on an update to my blog engine that had been sitting on my
to-do list for quite some time. This update brings improvements and new
features to w3blog.</p>
</span>
First of all, bloat was removed on the front end (client) side of things. I
got rid of Bootstrap, and JQuery, and rewrote the default CSS sheet. The
default look of the engine is actually pleasant to look at, the css sheet
weighs only 5.6K unminified (vs 138K of minified bootstrap's css), and of
course weight has also been slimmed down by removing JQuery (it was only
really needed by bootstrap).</p>
v0.4 default look</p>
</p>
v0.5 default look</p>
</p>
Speaking about improvements, comments now have a date of publishing. Really not
sure how I overlooked such a thing for so long.</p>
The new features are:</p>

Dynamic loading of posts on the main page, and category and archive pages. If
you turn on this feature, instead of getting the traditional pagination
buttons at the end of the page, you get a button to load the next batch of
posts through AJAX, so that it doesn't load a whole new page. You turn this
by adding 'dynamic_load': True</code> to WEBLOG_SETTINGS</code>.</li>
Infinite scroll for dynamic loading. When this and dynamic loading are
active, after scrolling to the bottom of the page (index, category page,
archive page), the next batch of posts are loaded automatically through AJAX.
Turn on dynamic loading (see above) and add 'infinite_load': True</code> to
WEBLOG_SETTINGS</code>.</li>
Floating sidebar. To turn this feature on, you add 'floating_sidebar': True</code>
to WEBLOG_SETTINGS</code>. What this does, is make the sidebar float next to the
content container after scrolling past the navbar and breadcrumb bar.</li>
These are basically all of the changes in this new release. If you happen to
already be using w3blog and are using a custom css sheet, you will need to
redo your css since the templates changed a bit since version 0.4, mainly the
html/css classes used. You will also need to make migrations and migrate them
to the database.</li>
</ul>
I had plans for other features for this release, but I decided to postpone them
for now. One of them was a way to add custom elements/html to the sidebar
without having to modify w3blog templates. Unfortunately I didn't find a sane
way of doing this.</p>
If you have any issues with w3blog, or maybe even suggestions, the first option
would be to open an issue on its Github page, if you don't have a Github
account (and don't want to create one), you can write me an email at
contact@yaroslavps.com</a>. However, if you are
using one of the big email providers (e.g. outlook or gmail), my response could
be blocked by your provider.</p>
w3blog's GitHub page</a>.</p>


Free and easy SSL with Let's Encrypt
2018-08-12T22:28:00+00:00
We all know that in this day and age security is an important factor when
developing sites, and not only. Also let's not forget about that sweet Google
SEO which rewards sites which use https instead of the old http (Although I
couldn't care less about Google, however some people I might work with do, and
I bet most of you reading also care).</p>
</span>
You might want to add SSL to your site, but, you might think, those SSL certs cost
money, and they ain't exactly cheap. And that's where you are wrong my friend
(unless you already knew about Let's Encrypt and are only reading this to learn
how to generate a certificate, in that case I apologize for the cringe
introduction).</p>
Here I will be detailing the steps to acquire an SSL certificate with Let's
Encrypt using Certbot, with nginx. I will be installing Certbot on a Debian
Stretch server, so you might need to input a different command depending on the
distro or OS you are using.</p>
First we need to install Certbot</p>
# apt-get install certbot
</span></code></pre>
Before we can install the certificates, we need to configure nginx for
certficate installation and renewal. We can set the default server
configuration on nginx for that. We need to do this so that Let's Encrypt's
server can read the challenge generated by Certbot and verify that we indeed
own the domain(s) for which we want to make the certificate.</p>
server {
</span>        listen 80 default_server;
</span>        listen [::]:80 default_server;
</span>
</span>        server_name _;
</span>
</span>        # This is so that Let's Encrypt can look for the challenge
</span>        location /.well-known/acme-challenge/ {
</span>                root /var/www/html/;
</span>                default_type text/plain;
</span>        }
</span>
</span>        # The rest of your configuration...
</span>}
</span></code></pre>
Once we have edited the default configuration file, we need to restart nginx.</p>
# sudo systemctl restart nginx
</span></code></pre>
After that we can get started installing the certificate. I wanted to get a
certificate for more than subdomain, so I added them using the -d flag to my
list of arguments like in the following example</p>
# certbot certonly --webroot -w /var/www/html -d www.example.com -d example.com
</span></code></pre>
It will ask your email for renew notifications, although certbot should renew
your certificates automatically before expiration.</p>
That's it, you should already have the new certificate and keys on your server.
Now you can go ahead and configure nginx to accept https connections and
redirect plain old insecure http requests to https. You should also put the
correct path for your certificate and public key under ssl_certificate</code> and
ssl_certificate_key</code> respectively.</p>
server {
</span>        listen 80;
</span>        listen [::]:80;
</span>        server_name www.example.com;
</span>        return 301 https://$host$request_uri;
</span>}
</span>
</span>server {
</span>        listen 443 ssl;
</span>
</span>        server_name www.example.com;
</span>        ssl_certificate /etc/letsencrypt/live/www.example.com/fullchain.pem;
</span>        ssl_certificate_key /path/to/key/www.example.com/privkey.pem;
</span>        location /static/ {
</span>                alias /home/webuser/mysite/static/;
</span>        }
</span>        location /media/ {
</span>                alias /home/webuser/mysite/media/;
</span>        }
</span>        location / {
</span>                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
</span>                proxy_set_header Host $http_host;
</span>                proxy_redirect off;
</span>                proxy_pass http://127.0.0.1:8001;
</span>        }
</span>}
</span></code></pre>
The downside of automatically redirecting http requests to https is that old
browsers that don't support https won't be able to open your site. But come on,
it's 2018, who cares about 10+ year old software. Even if you are running
really old hardware there definitely is some modern free software that runs on
that hardware.</p>
For more information on Certbot:
https://certbot.eff.org/docs/using.html</a></p>
Update (2018/11/02): Sometime in 2018 Let's Encrypt added the possibility to
generate a wildcard SSL certificate (e.g. *.example.com).</p>
To generate such a certificate we will need to download certbot from the git
repository</p>
$</span> git clone https://github.com/certbot/certbot.git
</span></code></pre>
Then we cd into certbot and run</p>
$</span> ./certbot-auto certonly</span> --manual </span>\
</span>> </span>--preferred-challenges=dns \
</span>> </span>--email email@example.com \
</span>> </span>--server https://acme-v02.api.letsencrypt.org/directory \
</span>> </span>--agree-tos \
</span>> </span>-d </span>*</span>.example.com
</span></code></pre>
For this you will need to add a TXT record to the DNS settings of your domain,
since this only works using the dns challenge. Before pressing enter make sure
that your record has been deployed.</p>
Do keep in mind that it's not currently possible to renew this kind of
certificate automatically.</p>


How to break pip, and how to repair it...
2018-08-03T22:00:00+00:00
Sometimes a program may suggest to you something, and you may following said
program's suggestion, because, why not, you trust the devs because they've done
an amazing job developing the program, and heck, they should know what they are
doing with their tools.</p>
</span>
With pip, this does not seem to be the case however. I've come across this
issue on my laptop and desktop computers running Manjaro, and while fixed it on
my laptop, I forgot how to fix it by the time I encountered on my desktop. The
problem: pip refuses to collect a package while trying to install any package
system wide (e.g. pywal), exiting with an exception like 'ModuleNotFoundError'
or something similar.</p>
First, I'll write down what NOT to do to avoid getting this error.</p>
You may get a message like this when after installing a package with pip</p>
You</span> are using pip version x.x.x, however version x.x is available.
</span>You</span> should consider upgrading via the </span>'</span>pip install --upgrade pip</span>'</span> command.
</span></code></pre>
If you get the following while NOT using a virtual environment, don't pay
attention to it. Don't even look at it. Just ignore it. I didn't, and now I am
writing a post about how to fix the mess that it causes because of following
that seemingly innocent and well-intended advice. At least not if you are using
an Arch-based distribution, don't really know about how this is managed in
other distributions, since I don't recall having this problem when using
Debian.</p>
Now, if you are using a virtual environment, you obviously can and should
upgrade pip this way. However, system-wide, your package manager (pacman)
should take care of upgrading pip along with your other packages when you
upgrade your system (pacman -Syu).</p>
The fix</h2>
If you unfortunately, like me, already ran the upgrade through pip, you should
first delete some files, by running this command:</p>
$</span> sudo rm</span> -rf</span> /usr/lib/python3.6/site-packages/pip /usr/lib/python3.6/site-packages/pkg_resources
</span></code></pre>
Then, you should reinstall pip and setup tools:</p>
$</span> sudo pacman</span> -S</span> python-setuptools python-pip
</span></code></pre>
And that's it! Your pip should now be as good as new.</p>
I found the solution in this forum thread: https://bbs.archlinux.org/viewtopic.php?id=237451</a></p>


w3blog, a simple blog engine
2018-07-23T02:05:00+00:00
</p>
In this post I will be explaining how to use my blog engine for the Django
framework, w3blog.</p>
</span>First, an introduction</h2>
I decided to create my own blog engine because I didn't find one for Django
that satisfied my needs. That is, I wanted a simple to setup, but customizable
blog engine that offered multilingual capabilities since I wanted to write my
blog in three different languages (English, Spanish and Russian). I wanted to
be able to write posts in different languages and make use of Django's built-in
localization engine, so that if user's locale is, for example, Spanish, so that
they would immediately see the posts in Spanish, if they are already translated
to Spanish.</p>
The following is a short list of features of w3blog:</p>

Users can post anonymous comments, if enabled</li>
Allow only registered users to post comments through Django's built-in
authentication engine, if enabled</li>
Can use own base template</li>
RSS</li>
Turn on or off the sidebar, and chose to show categories, archive list, none,
or both in the sidebar</li>
Show or hide the author of the post, and whether it shows the author's name
or username</li>
Enable or disable multilingual features</li>
</ul>
These are features available as of the writing of this post (v0.4.2). In the
next version (v0.5.x) I plan on adding the ability to further customize the
sidebar without having to modify the internal templates of the app.</p>
I will be showing you further down below how to setup the engine to customize these features.</p>
Setup</h2>
First of all, you need to install a couple of python modules with pip. w3blog
itself, and django-summernote. Check out https://summernote.org/ for more
information about it.</p>
$</span> pip install w3blog django-summernote
</span></code></pre>
After installing them, we need to add to the settings.py list of installed apps</p>
INSTALLED_APPS </span>= </span>[
</span>    </span>'</span>...</span>'</span>,
</span>    </span>'</span>django_summernote</span>'</span>,
</span>    </span>'</span>weblog</span>'</span>,
</span>]
</span></code></pre>
And add the following urls to our project's main urls.py</p>
url</span>(</span>r</span>'</span>^</span>blog/</span>'</span>, </span>include</span>(</span>'</span>weblog.urls</span>'</span>)),
</span>url</span>(</span>r</span>'</span>^</span>summernote/</span>'</span>, </span>include</span>(</span>'</span>django_summernote.urls</span>'</span>)),
</span></code></pre>
So if we now run our development server and go to our browser to the respective
link, we should now see something similar to the following</p>
</p>
After creating a django super user and going to the admin, we can now see a
weblog section with two models: Blog posts and Categories.</p>
</p>
We can add a category</p>
</p>
We could try and add a post now</p>
</p>
Chose a slug (the portion of the url for the post), the categor(y/ies), whether
we want it to be available to the reader after saving (Published) and the date
and time of publication.</p>
</p>
After adding our post, the page now looks like this</p>
</p>
I will better explain what the options above mean next.</p>
Configuration</h2>
That already looks much like a blog, but what if we wanted to use our own base
template (the one containing the header/navbar), or show the categories on the
sidebar. If you want to change the settings for w3blog, you need to add the
WEBLOG_SETTINGS</code> dictionary to your settings.py. The following are available
settings that you can change (with their defaults):</p>
WEBLOG_SETTINGS </span>= </span>{
</span>    </span>'</span>enable_comments</span>'</span>: </span>False</span>,
</span>    </span>'</span>allow_anon_comments</span>'</span>: </span>False</span>,
</span>    </span>'</span>multilingual</span>'</span>: </span>True</span>,
</span>    </span>'</span>blog_title</span>'</span>: </span>'</span>Django-Weblog</span>'</span>,
</span>    </span>'</span>base_template</span>'</span>: </span>'</span>weblog_base.html</span>'</span>,
</span>    </span>'</span>show_author</span>'</span>: </span>True</span>,
</span>    </span>'</span>use_authors_username</span>'</span>: </span>True</span>,
</span>    </span>'</span>show_sidebar</span>'</span>: </span>True</span>,
</span>    </span>'</span>show_categories</span>'</span>: </span>False</span>,
</span>    </span>'</span>show_archive</span>'</span>: </span>True</span>,
</span>    </span>'</span>posts_per_page</span>'</span>: </span>10</span>,
</span>    </span>'</span>enable_rss</span>'</span>: </span>True</span>,
</span>    </span>'</span>home_title</span>'</span>: </span>'</span>Welcome to the blog!</span>'</span>,
</span>}
</span></code></pre>
So most of them are pretty self explanatory, but I will explain them to avoid
confusion nonetheless:</p>

enable_comments -- set to True to enable users to post comments on the posts.</li>
allow_anon_comments -- allow anonymous comments.</li>
multilingual -- enable multilingual features, i.e. translations of posts.</li>
blog_title -- the title of the blog, will be used on the blog's homepage and
title tag</li>
base_template -- which base template to use. If not set, will use w3blog's
default template.</li>
show_author -- set to True to display the author of the post on each post</li>
use_authors_username -- set to True to use the author's username (e.g.
author66) instead of their full name (e.g. John Smith).</li>
show_sidebar -- set to True to show the sidebar (where the category and
archive lists reside).</li>
show_categories -- set to True to show the categories list on the sidebar
(won't show if you disabled the sidebar)</li>
show_archive -- set to True to show the archive list on the sidebar (won't
show if you disabled the sidebar)</li>
posts_per_page -- the number of pages to display per page.</li>
enable_rss -- set to True to enable the RSS feed (/rss)</li>
home_title -- if set, it will display on the blog's home page instead of
blog_title (but blog_title will display in the title tag).</li>
</ul>
Now, you most probably will want to use your own template, in that case, make
sure to add the following to your template's head tag</p>
<meta charset="utf-8">
</span><meta content="width=device-width, initial-scale=1" name="viewport">
</span><title>{{ </span>blog_title </span>}} - {% </span>block </span>title_block </span>%} Home {% </span>endblock </span>%}</title>
</span><link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css" integrity="sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4" crossorigin="anonymous">
</span><link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.0.13/css/all.css" integrity="sha384-DNOHZ68U8hZfKXOrtjWvjxusGo9WQnrNx2sqG0tfsghAvtVlRW3tvkXWZh58N9jp" crossorigin="anonymous">
</span><link rel="stylesheet" href="{% </span>static </span>'</span>/weblog/css/weblog.css</span>' </span>%}">
</span><script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
</span><script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.0/umd/popper.min.js" integrity="sha384-cs/chFZiN24E4KMATLdqdvsezGxaGsi4hLGOzlXwp5UZB1LY//20VyM2taTB4QvJ" crossorigin="anonymous"></script>
</span><script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/js/bootstrap.min.js" integrity="sha384-uefMccjFJAIv6A+rW+L4AHf99KvxDjWSu1z9VI8SKNVmz4sk7buKt/6v9KI65qnm" crossorigin="anonymous"></script>
</span><script src="{% </span>static </span>'</span>/weblog/js/weblog.js</span>' </span>%}" defer></script>
</span></code></pre>
Another option, if you don't want to use bootstrap, would be to use your own
css files using bootstrap's classes. You will want to check the templates'
source code of my engine for reference in w3blog's GitHub
page</a>.</p>
Using w3blog</h2>
Let us go back a little bit and discuss how to use w3blog. As we saw in the
beginning, you can add posts and categories through the admin. Let's focus on
the adding posts part by checking this screenshot</p>
As you can see, there are several option there to chose from</p>

Preview image -- use this to upload an image that you want to be displayed in
the preview of your post (in the home/category/archive page)</li>
Preview text -- use this to set the text that will be displayed in the
preview of your post. If you don't set it, w3blog will automatically use the
first paragraph of your post's body as preview text.</li>
Original language (ISO) -- the ISO code of the original language of your post
(e.g. "en"). This setting is not mandatory, and if you don't set and have
translations for your post, it will assume that the original language is the
one set in your settings.py file (LANGUAGE variable).</li>
Slug (URL) -- this will be the url part of your post. If you still don't
understand what a slug is, Check out this link.</li>
Categories -- the category or categories that your post will belong to. If
none are set, your post will be categorized under an "Uncategorized"
category. Important note here: you should not set the slug of a category to
"misc" since that is already used by weblog to display posts without a
category.</li>
Pin blog post -- use this if you want to pin the post to the home page and
show above the rest of the posts</li>
Pinned post priority -- if you pinned several posts, you use this to set
their priority</li>
Published -- this sets whether the post is available to your readers. This
can be useful, for example, when you've started writing a post, but haven't
finished and want to save a draft of it, so you don't check this checkbox so
that it doesn't display in your blog.</li>
Publish date -- date and time the post was published.</li>
</ul>
Let's try making a new post and setting it to pinned, to see what happens</p>
</p>
So it will appear in our homepage at the very beginning</p>
</p>
Translations</h2>
Adding a translation to a post is really simple if we have the multilingual
feature enabled in the settings. While adding our post, or after adding it, we
can go to the bottom of the add blog post page and add it. We will see some
similar options to the ones for the original blog post.</p>
Let's suppose I want to add a Spanish translation. For that, I would type in
"es" as Language, and then just type the title, content, and if needed, the
preview image and text for my Spanish translation.</p>
</p>
After saving, if we go to the blog with computer or browser with Spanish
language locale set, we will the site in spanish with the aforementioned's post
Spanish translation</p>
</p>
Or if we go to the post itself, we will see an option to choose to read the option in a different language.</p>
</p>
Similarly, we can add translations for each category, if we so wish.</p>
So, this is basically it on how to use w3blog to set up your own blog using
Django. I hope this post and my blog engine were useful to you. If you have any
questions you can leave a comment here, or</del> contact me at my email
contact@yaroslavps.com</a></p>
If you have found a bug or problem with w3blog, be sure to open an issue on
GitHub. The app is currently only translated to English, Spanish, and Russian,
so if you are willing to provide a translation for your language or any other
language that you know, you can send a pull request with the translated
strings, or download the strings for the git repository, translate them, and
send them to me so that I can compile them and add them to the project myself.</p>
Thank you very much for reading this post, and I will be very happy to see your
projects using my blog engine!</p>


Building and deploying a personal site with blog. Part 2
2018-07-21T22:26:00+00:00
Welcome back, this post is a continuation of the previous post in which I wrote
about how to initialize and setup a Django project for a personal site with
blog locally. Now we will be focusing on the juicy part, getting your site
deployed so that everyone on the interwebz can get to see your nice new dank
memes in your very own site!</del> the fruits of your sweat and blood.</p>
</span>
You might want to read part one</a></em></p>
Part II: Deploying your site</h2>
I will assume that you already have bought a VPS or dedicated server in any of
the many hosting services available online, or maybe if you are cool enough,
might even have a machine of your own with a properly setup static IP address.
Note that if you are using shared hosting, the deployment process is usually
already automatized if they have support for Python and Django, if they don't
support Python and Django, you need to get another hosting solution.</p>
If you still don't have any kind of hosting, and you are looking for a nice and
comfy VPS for your projects, I would recommend using DigitalOcean, you can get
a VPS powerful enough for Django and other Python web projects for as low as
$5. You can follow this referral link if you have decided to try DigitalOcean,
it will give a $10 credit for your first account, enough for two whole months
on their most inexpensive VPS.</p>
Once you have your server set and ready to go, we can continue with the fun
part. Deploying your site. I will be assuming that you have a Debian-based
distro in your server.</p>
First steps</h3>
First let's make sure that our server is up to date</p>
$</span> sudo apt-get update
</span>$</span> sudo apt-get upgrade
</span></code></pre>
Next, we need to install all the needed packages</p>
$</span> sudo apt-get install python3 python3-pip python-virtualenv postgresql nginx supervisor
</span></code></pre>
You can set the virtual environment with your login user, or create another
user exclusively to run your Django project and other related stuff.</p>
You can add a user to run the app like so</p>
$</span> adduser webuser
</span></code></pre>
Let's go ahead and create a directory for virtual environments and create the environment for our project</p>
$</span> virtualenv mysite</span> -p</span> python3
</span></code></pre>
Now we source the environment</p>
$</span> source mysite/bin/activate
</span></code></pre>
And use pip to install the necessary modules</p>
$</span> pip install Django Pillow psycopg2 django-summernote w3blog
</span></code></pre>
After that, we start and enable postgresql</p>
$</span> sudo systemctl start postgresql
</span>$</span> sudo systemctl enable postgresql
</span></code></pre>
Login to the postgres shell, set the password for user postgres, and create the
database for our project</p>
$</span> sudo su - postgres
</span>$</span> psql
</span>postgres</span>=</span># </span>\p</span>assword
</span>postgres</span>=</span># </span>create</span> database mysitedb</span>;
</span></code></pre>
In this case, I am using the postgres user directly since I am using this
server just for myself, and I only use PSQL for my blog. However, if you host
multiple projects, and/or multiple users have a more direct access to postgres,
you might want to create separate users and roles for each database or group of
databases.</p>
Copying your Django project to the server</h3>
One way to copy your project files to the server would be using scp, for
example</p>
scp</span> -r</span> /path/to/project user@serveripordomain:/home/webuser/
</span></code></pre>
Or if you are already hosting your project as a git repository in a service
like GitHub or BitBucket, you could clone directly into you repository in the
home folder, like so</p>
$</span> git clone https:/github.com/user/repo
</span></code></pre>
After the files have been copied, we need to set some things up for our Django
project. Before running this commads, you might need to export the needed
environmental variables, if you set up your Django settings using environmental
variables as I wrote in the first part. Otherwise, you might want to edit your
settings.py file now with the correct settings.</p>
First we collect static files</p>
$</span> ./manage.py collectstatic
</span></code></pre>
Then, we migrate our models to the database</p>
$</span> ./manage.py migrate
</span></code></pre>
And after migrating we create a superuser for the Django admin</p>
$</span> ./manage.py createsuperuser
</span></code></pre>
If you get an authentication error while performing any of the above commands,
you will have to edit the following file</p>
$</span> sudoedit /etc/postgresql/9.1/main/pg_hba.conf
</span></code></pre>
Change this line</p>
local        all        postgres        peer
</span></code></pre>
So that it looks like this</p>
local        all        postgres        md5
</span></code></pre>
Configuring the server</h3>
I will be assuming that you are going to use nginx as a proxy server for
Django. There are other alternatives such as Apache, but I found nginx to be a
really powerful and easy to setup server.</p>
First, while in our virtual environment, we will install gunicorn. We might
have been using Django's own development server while building and testing our
app locally, but that's what it is, a development server. It is not mean for
production, so we will use gunicorn for that</p>
pip</span> install gunicorn
</span></code></pre>
Now we'll cd into our env's bin folder and create a file called
gunicorn_start</code>, and add the following lines to it:</p>
#!/bin/bash
</span>NAME</span>=</span>"</span>mysite</span>"
</span># Our site's directory
</span>DIR</span>=</span>/home/webuser/mysite
</span>USER</span>=</span>webuser
</span>GROUP</span>=</span>webuser
</span>WORKERS</span>=</span>3
</span>BIND</span>=</span>127.0.0.1:8001
</span>DJANGO_SETTINGS_MODULE</span>=</span>mysite.settings
</span>DJANGO_WSGI_MODULE</span>=</span>mysite.wsgi
</span>LOG_LEVEL</span>=</span>error
</span>
</span>cd </span>$</span>DIR
</span>source</span> /home/webuser/venvs/mysite/bin/activate
</span>
</span>export </span>DJANGO_SETTINGS_MODULE</span>=</span>$</span>DJANGO_SETTINGS_MODULE
</span>export </span>PYTHONPATH</span>=</span>$</span>DIR</span>:</span>$</span>PYTHONPATH
</span>
</span># This are the environment variables needed for your Django settings
</span># (see the first part of this guide)
</span>export </span>SITE_SECRETKEY</span>=</span>"</span>YOURSECRETKEYGOESHERE</span>"
</span>export </span>SITE_DEBUG</span>=</span>"</span>false</span>"
</span>export </span>SITE_DBPASSWORD</span>=</span>"</span>YOURPOSTGRESPWDGOESHERE</span>"
</span>export </span>SITE_DBUSER</span>=</span>"</span>postgres</span>"
</span>export </span>SITE_HOST</span>=</span>"</span>localhost</span>"
</span>export </span>SITE_PORT</span>=</span>"</span>5342</span>"
</span>
</span>exec</span> /home/webuser/venvs/mysite/bin/gunicorn </span>$</span>{</span>DJANGO_WSGI_MODULE</span>}:application \
</span>  --name </span>$</span>NAME </span>\
</span>  --workers </span>$</span>WORKERS </span>\
</span>  --user=</span>$</span>USER </span>\
</span>  --group=</span>$</span>GROUP </span>\
</span>  --bind=</span>$</span>BIND </span>\
</span>  --log-level=</span>$</span>LOG_LEVEL </span>\
</span>  --log-file=-
</span></code></pre>
And make it executable</p>
$</span> chmod a+x gunicorn_start
</span></code></pre>
That's almost it, but before actually running gunicorn, we will install one
more program which will take care of the gunicorn server, so that if for any
reason we need to reboot our server, it will restart gunicorn for us. It needs
to be some kind of supervisor. Well why my dear friend, we are going to use
Supervisor.</p>
But first, let's create a folder name logs inside our webuser's home folder,
and create a file to be used to log any application errors:</p>
$</span> mkdir logs
</span>$</span> touch logs/gunicorn.log
</span></code></pre>
Now we create a new Supervisor config file:</p>
$</span> sudoedit /etc/supervisor/conf.d/mysite.conf
</span></code></pre>
Add the following:</p>
[program:mysite]
</span>command</span>=/</span>home</span>/</span>webuser</span>/</span>venvs</span>/</span>mysite</span>/</span>bin</span>/</span>gunicorn_start
</span>user</span>=</span>webuser
</span>autostart</span>=</span>true
</span>autorestart</span>=</span>true
</span>redirect_stderr</span>=</span>true
</span>stdout_logfile</span>=/</span>home</span>/</span>webuser</span>/</span>logs</span>/</span>gunicorn</span>.</span>log
</span></code></pre>
We reread the Supervisor configuration files, and start the program:</p>
$</span> sudo supervisorctl reread
</span>$</span> sudo supervisorctl update
</span></code></pre>
And check the status to make sure it is running fine and dandy</p>
$</span> sudo supervisorctl status mysite
</span></code></pre>
Next we should configure nginx, and for that we are going to make the
configuration file for our site. Open it up</p>
$</span> sudoedit /etc/nginx/sites-available/mysite
</span></code></pre>
And add the following lines</p>
server {
</span>        listen 80;
</span>        listen [::]:80;
</span>
</span>        server_name www.example.com;
</span>
</span>        location /static/ {
</span>                alias /home/webuser/mysite/static/;
</span>        }
</span>
</span>        location /media/ {
</span>                alias /home/webuser/mysite/media/;
</span>        }
</span>
</span>        location / {
</span>                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
</span>                proxy_set_header Host $http_host;
</span>                proxy_redirect off;
</span>                proxy_pass http://127.0.0.1:8001;
</span>        }
</span>}
</span></code></pre>
I won't be explaining in this guide how to setup an SSL certificate, since it
is out of its scope, but, if you want to get a free certificate, I cant point
you to letsencrypt.com. It is backed by many of the top internet organizations.
If you already have an SSL certificate, and would like to use HTTPS, your file
should look like this</p>
server {
</span>        listen 80;
</span>        listen [::]:80;
</span>        server_name www.example.com;
</span>        return 301 https://$host$request_uri;
</span>}
</span>
</span>server {
</span>        listen 443 ssl;
</span>
</span>        server_name www.example.com;
</span>        ssl_certificate /path/to/cert/fullchain.pem;
</span>        ssl_certificate_key /path/to/key/privkey.pem;
</span>        location /static/ {
</span>                alias /home/webuser/mysite/static/;
</span>        }
</span>        location /media/ {
</span>                alias /home/webuser/mysite/media/;
</span>        }
</span>        location / {
</span>                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
</span>                proxy_set_header Host $http_host;
</span>                proxy_redirect off;
</span>                proxy_pass http://127.0.0.1:8001;
</span>        }
</span>}
</span></code></pre>
After creating the needed config file, we create a symbolic link to it in sites-enabled</p>
$</span> sudo ln</span> -s</span> /etc/nginx/sites-available/mysite /etc/nginx/sites-enabled/mysite
</span></code></pre>
Remove the default nginx site</p>
$</span> sudo rm /etc/nginx/sites-enabled/default
</span></code></pre>
Finally, we restart nginx</p>
$</span> sudo service nginx restart
</span></code></pre>
Et voila!</p>
</p>
Final thoughts</h2>
This was by no means the most comprehensive guide to Django, but it might help
you get started on setting up your own projects, and most importantly, actually
deploying them to a live server.</p>


Building and deploying a personal site with blog. Part 1
2018-07-21T21:46:00+00:00
I will be explaining in this guide how I built, and how I deployed this site.
This will be a really simple tutorial, on how to do it using Python and Django.
You might want to have a little knowledge of Django and Python previous to
attempting to follow the instructions laid out in this tutorial, as the goal of
this tutorial is not to teach you the basic principles of either the Python
programming language, or Django framework. It is just meant as a little guide
on the basic workflow when building and deploying sites with Django.</p>
</span>
Initially I intended for the whole guide to be just one post, but I had to
break it into two posts since it ended up being too long. This first part will
focus on the initial process of setting up and building a kind of personal blog
site based on what I did for my own site, using the Django framework.</p>
I will not be writing here on how I made the blog engine for this site, rather
I will be using a blog engine that I have made, called w3blog (you can read it
either as "weblog", or as "w3/www blog"). Checkout the GitHub page
here</a>. I will not be writing on this
guide all of the possible configurations of w3blog, I will be writing a
separate post about it soon (actually, I should also write thorough
documentation for the module, and I will, but while I get enough time to do it,
I will write a short post about it in the meantime).</p>
So let's get our hands dirty...</p>
Part I: Building and getting the site ready for deployment</h2>
Before getting started, we need to make that everything we need is installed.
In this case I am building my site using Python 3.6, and Django 2.0. I am also
using PostgreSQL as my database.</p>
First steps</h3>
I am using Manjaro as my OS, so I already have installed Python 3.6. You should
also have Python 3.6 installed if you are using Ubuntu 18.04. PostgreSQL
doesn't come by default, so using the terminal I installed postgres using the
following command (this is for Manjaro and other Arch based distros):</p>
$</span> sudo pacman</span> -S</span> postgresql
</span></code></pre>
Or for Ubuntu and other Debian-based distros:</p>
$</span> sudo apt-get update
</span>$</span> sudo apt-get install postgresql
</span></code></pre>
If you're using Windows, these commands will obviously not be of much help
(unless you are using the Linux Subsystem for Windows, or whatever Micro$oft is
calling it these days).</p>
I would recommend using Linux for web development (or any kind of development
(or any kind of use case for that matter)), especially with Python, or even Mac
OSX (or whatever Apple has taken to calling their OS these days), since it is
also Unix-like, and you have bash installed natively. However, if you do plan
on using Windows to develop your site, you should be able to download the
respective installers from their respective websites.</p>
Anyway, back to the topic.</p>
Next, we start PostgreSQL using systemd (or your init system of choice):</p>
$</span> sudo systemctl start postgresql
</span></code></pre>
Once we have Python and Postgres installed, we need to create a virtual
environment to keep all the packages that we need for our site all in one
place. Now, you could of course skip this step and just install the packages
system-wide, however, it is a better practice to have different environments
for different projects.</p>
I am not going to be explaining the reasons for that since there already are a
lot of articles on the internetz which do a better job at explaining why, than
I would.</p>
If you don't have virtualenv installed, go ahead and do it now:</p>
$</span> sudo pacman</span> -S</span> python-virtualenv
</span></code></pre>
Or</p>
$</span> sudo apt-get install python-virtualenv
</span></code></pre>
Now we cd into the directory where we want to keep our environments and do:</p>
$</span> virtualenv envname</span> -p</span> python3
</span></code></pre>
After some seconds we should have a new folder with our new virtual
environment. To activate the environment we need to source the "activate"
script inside the environment, like so:</p>
$</span> source envname/bin/activate
</span></code></pre>
You should now notice the name of the environment on the left side of your
prompt. After that we can proceed to install with pip the packages that we
need, for example, to install Django:</p>
$</span> pip install Django
</span></code></pre>
The list of packages needed to build the site in this tutorial:</p>
Django
</span>Pillow
</span>psycopg2
</span>django-summernote
</span>w3blog
</span></code></pre>
Pillow is a package used by Django to work with images saved on the database.
psycopg2 is needed to use PostgreSQL with and Django. w3blog is my blog engine,
it makes use of django-summernote, a module to integrate the wonderful editor
summernote. You can find out more about it here</a>.</p>
Starting and setting up the project</h3>
After that we are ready to initialize our project. We cd into a directory where
we usually keep all our projects, and tell django to initialize it for us</p>
$</span> django-admin startproject mysite
</span></code></pre>
Django will create the basic directory structure for our project and generate
some basic files to get started. If you run "manage.py runserver" right now you
will get the welcome screen from Django telling you your project has been
correctly initialized. I will be skipping that part and go on ahead and start
configuring my project as needed.</p>
First let's set up the settings.py</code> file by modifying the following settings,
the other we can leave for the time being:</p>
from </span>django.utils.translation </span>import </span>ugettext_lazy </span>as </span>_
</span>
</span>SECRET_KEY </span>= </span>os.environ.</span>get</span>(</span>'</span>SITE_SECRETKEY</span>'</span>)
</span>DEBUG </span>= </span>os.environ.</span>get</span>(</span>'</span>SITE_DEBUG</span>'</span>) </span>== </span>true
</span>
</span>DATABASES </span>= </span>{
</span>    </span>'</span>default</span>'</span>: {
</span>        </span>'</span>ENGINE</span>'</span>: </span>'</span>django.db.backends.postgresql</span>'</span>,
</span>        </span>'</span>NAME</span>'</span>: </span>'</span>sitedb</span>'</span>,
</span>        </span>'</span>USER</span>'</span>: os.environ.</span>get</span>(</span>'</span>SITE_DBUSER</span>'</span>),
</span>        </span>'</span>PASSWORD</span>'</span>: os.environ.</span>get</span>(</span>'</span>SITE_DBPASSWORD</span>'</span>),
</span>        </span>'</span>HOST</span>'</span>: os.environ.</span>get</span>(</span>'</span>SITE_DBHOST</span>'</span>),
</span>        </span>'</span>PORT</span>'</span>: os.environ.</span>get</span>(</span>'</span>SITE_DBPORT</span>'</span>)
</span>    }
</span>}
</span>
</span>LANGUAGE_CODE </span>= </span>'</span>en-us</span>'
</span>
</span>LANGUAGES </span>= </span>(
</span>        (</span>'</span>es</span>'</span>, </span>_</span>(</span>'</span>Spanish</span>'</span>)),
</span>        (</span>'</span>en</span>'</span>, </span>_</span>(</span>'</span>English</span>'</span>)),
</span>        (</span>'</span>ru</span>'</span>, </span>_</span>(</span>'</span>Russian</span>'</span>)),
</span>)
</span>
</span>LOCALEPATHS </span>= </span>(
</span>os.path.</span>join</span>(</span>BASE_DIR</span>, </span>'</span>locale</span>'</span>),
</span>)
</span>
</span>TIME_ZONE </span>= </span>'</span>Europe/Moscow</span>'
</span>
</span>USE_I18N </span>= </span>True
</span>USE_L10N </span>= </span>True
</span>USE_TZ </span>= </span>True
</span>
</span>STATIC_URL </span>= </span>'</span>/static/</span>'
</span>MEDIA_URL </span>= </span>'</span>/media/</span>'
</span>MEDIA_ROOT </span>= </span>os.path.</span>join</span>(</span>BASE_DIR</span>, </span>'</span>media/</span>'</span>)
</span></code></pre>
You may have noticed that I am using os.environ.get('ENVVAR_NAME')</code> to get
some of my settings from environmental variables, instead of typing those
settings directly into my file. Part of the reason I do that, is because it is
not safe to store information such as passwords, or secret keys in your code.</p>
Also, since not all of my settings are the same in my local, or testing
machine, as in my server or production machine, this makes it more convenient
and easier, since I don't have to keep two different "settings.py" files, and I
don't risk the chance of, for example, uploading my passwords, and other
private information to a service like GitHub, or BitBucket, should I decide to
publicly host the source code of my site.</p>
There are other reasons for that, and there are other ways to keep config
separated from code, but I will not be going into much detail now.</p>
To export the environmental variables while you're developing, you can create a
script (for example variables.sh) that will export them for you, it should
something like this</p>
#!/bin/sh
</span>export </span>SITE_SECRETKEY</span>=</span>"</span>YOURSECRETKEYGOESHERE</span>"
</span>export </span>SITE_DEBUG</span>=</span>"</span>false</span>"
</span>export </span>SITE_DBPASSWORD</span>=</span>"</span>YOURPOSTGRESPWDGOESHERE</span>"
</span>export </span>SITE_DBUSER</span>=</span>"</span>postgres</span>"
</span>export </span>SITE_HOST</span>=</span>"</span>localhost</span>"
</span>export </span>SITE_PORT</span>=</span>"</span>5342</span>"
</span></code></pre>
To export the variables from the file to your current shell you would run
either</p>
$</span> source variables.sh
</span></code></pre>
or</p>
$</span> . variables.sh
</span></code></pre>
Both work equally fine.</p>
The other settings are the language and locale settings. This is an example of
my settings, so you should them accordingly to your needs. For example, my site
is available in three languages, so I them accordingly in the LANGUAGES</code>
tuple, and also enabled Django's localization services by setting the
USE_I18N</code> and USE_L10N</code> variables to True.</p>
If you want Django to translate your site according to the user settings
automatically, you also need to add the following to MIDDLEWARE</code> before
CommonMiddleWare</code>, but after SessionMiddleware</code>:</p>
    </span>'</span>django.middleware.locale.LocaleMiddleware</span>'</span>,
</span></code></pre>
The last three settings back there, are so that Django knows where to look for
static files such as images, css, js, etc. And so that it knows where to save
and look for images belonging to models in the database, for example, if a user
uploads a picture, or this case, if I embed an image in one of my blog posts.</p>
Let us not forget about adding the modules that we downloaded with pip to the list of installed apps:</p>
INSTALLED_APPS </span>= </span>[
</span>    </span>'</span>...</span>'</span>,
</span>    </span>'</span>django-summernote</span>'</span>,
</span>    </span>'</span>weblog</span>'</span>,
</span>]
</span></code></pre>
And set our allowed hosts:</p>
ALLOWED_HOSTS </span>= </span>[
</span>    </span>'</span>127.0.0.1</span>'</span>,
</span>    </span>'</span>mysite.xyz</span>'</span>,
</span>]
</span></code></pre>
Configuring the DB</h3>
With these settings, we can now migrate our models to the database. However,
before doing that we need to set up the database in PostgreSQL. For that we
need to open up the terminal and switch to user postgres, and open up the psql
shell:</p>
$</span> sudo su - postgres
</span>$</span> psql
</span></code></pre>
Inside the shell, if you haven't set up a password for user postgres, do so
now:</p>
postgres</span>=</span># </span>\p</span>assword
</span></code></pre>
After that, we create our database</p>
postgres</span>=</span># </span>create</span> database sitedb</span>;
</span></code></pre>
And NOW, we can migrate our models. Let us do so by going back to our project's
root directory, and typing:</p>
$</span> ./manage.py migrate
</span></code></pre>
If for some reason, it complains to you about some model migrations conflicts,
go ahead and do the following, typing "y" when it prompts you:</p>
$</span> ./manage.py makemigrations</span> --merge
</span></code></pre>
After migrating we can create a superuser so that we can enter the Django admin
site:</p>
# ./manage.py createsuperuser
</span></code></pre>
Launching the project</h3>
One final thing before our site is ready for use. We need to add the
corresponding urls for the w3blog engine, and summernote editor in our projects
main urls.py</code> file. So let's go ahead and add them, our urls.py file should like
this:</p>
from </span>django.contrib </span>import </span>admin
</span>from </span>django.conf </span>import </span>settings
</span>from </span>django.urls </span>import </span>path, include
</span>from </span>django.conf.urls.static </span>import </span>static
</span>
</span>urlpatterns </span>= </span>[
</span>    </span>path</span>(</span>'</span>admin/</span>'</span>, admin.site.urls),
</span>    </span>path</span>(</span>'</span>summernote/</span>'</span>, </span>include</span>(</span>'</span>django_summernote.urls</span>'</span>)),
</span>    </span>path</span>(</span>''</span>, </span>include</span>(</span>'</span>weblog.urls</span>'</span>)),
</span>]
</span>if </span>settings.</span>DEBUG</span>:
</span>    urlpatterns </span>+= </span>static</span>(settings.</span>MEDIA_URL</span>, </span>document_root</span>=</span>settings.</span>MEDIA_ROOT</span>)
</span></code></pre>
Now we if we launch our development server</p>
$</span> ./manage.py runserver
</span></code></pre>
And go to 127.0.0.1:8000</code> in our browser, we should see a basic blog site with
no posts on it yet, however, if we go to the django admin page, we can some
posts, and so our site is not so empty anymore!</p>
Now, if you are planning on having some other things on you site besides the
blog, you need to start a new django app inside your project's root directory</p>
$</span> django-admin startapp myapp
</span></code></pre>
Add an include to it's urls config to the main urls.py file. So your project's
urlpatterns variable now might be looking something like this:</p>
urlpatterns </span>= </span>[
</span>    </span>path</span>(</span>'</span>admin/</span>'</span>, admin.site.urls),
</span>    </span>path</span>(</span>'</span>summernote/</span>'</span>, </span>include</span>(</span>'</span>django_summernote.urls</span>'</span>)),
</span>    </span>path</span>(</span>'</span>weblog/</span>'</span>, </span>include</span>(</span>'</span>weblog.urls</span>'</span>)),
</span>    </span>path</span>(</span>''</span>, </span>include</span>(</span>'</span>myapp</span>'</span>)),
</span>]
</span></code></pre>
And don't forget to add your app's name to the INSTALLED_APPS</code> list in
"settings.py".</p>
This is it for part 1 of this little guide. Soon I will post the second part where I'll write about how I deployed my site to a Debian server.</p>
Continue reading part two here</a>.</p>


A weblog by Yaroslav de la Peña Smirnov
2018-07-21T20:06:00+00:00
Hello, and welcome to my weblog. As you may have read in my home page, I will be
writing here mainly about what I do, and how I do it. That is, I aim to make a
sort of compendium for myself as a self-learning tool; for future reference,
when I need to refresh my memory, and of course for anybody else that shares my
interests and wishes to learn something new. I am by no means an expert, but I
believe that it is easier to learn by sharing your knowledge.</p>
</span>
I will be writing mainly in English, but I also plan on making most (probably
all of them) of my posts available in Spanish and Russian.</p>
Grab a cup of tea (or any kind of refreshment you might prefer), sit down, and
feel free to browse through my weblog.</p>
On my next post I will write about the steps that I took to build this site and
deploy it. So if you are interested in building sites using Django and Python as
I am, you might want to check out the next post when it comes out. You can also
add this blog's RSS feed to your favorite reader so you don't miss anything.</p>
Note: While my blog isn't so visited I plan on allowing anonymous (i.e. without
having to register) comments. If I start to get a lot of spammers, I will have
to turn them off until I integrate a better system for comments. So feel free to
leave your comments! All comments and suggestions are welcome (as long as they
are kept in a civilized manner).</p>

Modern vs old</h2>
</p>
Now that I have reviewed each computer separately I will be comparing them in each category I've reviewed them to see which is the better one overall.</p>

Tabs</h2>
Now I'll list the main arguments for and against using tabs. This list mostly mirrors the one on spaces, just from another point of view:</p>

Some links that might be of interest:</h3>

https://baymard.com/blog/line-length-readability</a></li>
https://www.paulolyslager.com/optimal-text-layout-line-length/</a></li>
http://bettermotherfuckingwebsite.com/</a></li> </ul>

Final thoughts</h2>
This was by no means the most comprehensive guide to Django, but it might help you get started on setting up your own projects, and most importantly, actually deploying them to a live server.</p>

A weblog by Yaroslav de la Peña Smirnov

Introduction to Generic Netlink, or How to Talk with the Linux Kernel

Some theory</h2>
As I've already mentioned, Netlink works over the usual BSD sockets. A Netlink message always starts with a Netlink header, followed by a protocol header, that is in the case of Generic Netlink, the Generic Netlink header.</p>

Overcomplicated

Liberating my X200

It's time to use crypto

How I debug programs in Vim

Why I don't use social media

New computers suck

Fixing terminal and remote machine misunderstandings

A Viral Year

Simple and Minimal Git Server Setup

A little fun with Zoom

The indentation Red Pill

Goodbye OpenVPN, hello Wireguard

Website refactor

Rethinking my weblog

Packaging and distributing python apps and modules

Intro to Linux and the Bash command line, pt III

Filters</h2>
These programs that perform operations on input text and then write them to standard output are commonly known as filters. You may already be familiar with one of these commands, which the first one that I'm going to talk about.</p>

Regular expressions</h2>
A regular expression, or regex for short, is a string of text, that define a search pattern for a larger set of text. Regexes are used in many programs, such as in text editors, and search engines, and can be also of great use in the terminal</p>

Post scriptum</h2>
Before ending it for good, I want to show some other programs that might be of use in the Bash command line.</p>

Intro to Linux and the Bash command line, pt II

File manipulation</h2>
I didn't get into detail in the previous part of this guide on the file manipulation side of things. However it is a crucial part of any computer system, after all they wouldn't be as useful as they are if you couldn't create, edit and delete files.</p>

Intro to Linux and the Bash command line

Setting up your own VPN

Some thoughts on web design

New w3blog release, version 0.5

Free and easy SSL with Let's Encrypt

How to break pip, and how to repair it...

w3blog, a simple blog engine

Building and deploying a personal site with blog. Part 2

Building and deploying a personal site with blog. Part 1

A weblog by Yaroslav de la Peña Smirnov

A weblog by Yaroslav de la Peña Smirnov

Introduction to Generic Netlink, or How to Talk with the Linux Kernel

Some theory</h2> As I've already mentioned, Netlink works over the usual BSD sockets. A Netlink message always starts with a Netlink header, followed by a protocol header, that is in the case of Generic Netlink, the Generic Netlink header.</p>

Netlink messages</h3> The headers look like this:</p> </p>

Overcomplicated

Liberating my X200

It's time to use crypto

How I debug programs in Vim

Why I don't use social media

New computers suck

Modern vs old</h2> </p> Now that I have reviewed each computer separately I will be comparing them in each category I've reviewed them to see which is the better one overall.</p>

Fixing terminal and remote machine misunderstandings

A Viral Year

Simple and Minimal Git Server Setup

A little fun with Zoom

The indentation Red Pill

Tabs</h2> Now I'll list the main arguments for and against using tabs. This list mostly mirrors the one on spaces, just from another point of view:</p>

Cons</h3> They are not displayed consistently. Once more, this argument doesn't really hold water for reasons I'll mention later.</li> Most people and projects use spaces for indentation, which means tabs are not the preferable indentation method. Once more, argumentum ad populum</em>.</li>

Goodbye OpenVPN, hello Wireguard

Website refactor

Rethinking my weblog

Packaging and distributing python apps and modules

Intro to Linux and the Bash command line, pt III

Filters</h2> These programs that perform operations on input text and then write them to standard output are commonly known as filters. You may already be familiar with one of these commands, which the first one that I'm going to talk about.</p>

Regular expressions</h2> A regular expression, or regex for short, is a string of text, that define a search pattern for a larger set of text. Regexes are used in many programs, such as in text editors, and search engines, and can be also of great use in the terminal</p>

Post scriptum</h2> Before ending it for good, I want to show some other programs that might be of use in the Bash command line.</p>

Intro to Linux and the Bash command line, pt II

File manipulation</h2> I didn't get into detail in the previous part of this guide on the file manipulation side of things. However it is a crucial part of any computer system, after all they wouldn't be as useful as they are if you couldn't create, edit and delete files.</p>

Intro to Linux and the Bash command line

Setting up your own VPN

Some thoughts on web design

New w3blog release, version 0.5

Free and easy SSL with Let's Encrypt

How to break pip, and how to repair it...

The fix</h2> If you unfortunately, like me, already ran the upgrade through pip, you should first delete some files, by running this command:</p>

w3blog, a simple blog engine

Some theory</h2>
As I've already mentioned, Netlink works over the usual BSD sockets. A Netlink message always starts with a Netlink header, followed by a protocol header, that is in the case of Generic Netlink, the Generic Netlink header.</p>

Modern vs old</h2>
</p>
Now that I have reviewed each computer separately I will be comparing them in each category I've reviewed them to see which is the better one overall.</p>

Tabs</h2>
Now I'll list the main arguments for and against using tabs. This list mostly mirrors the one on spaces, just from another point of view:</p>

Filters</h2>
These programs that perform operations on input text and then write them to standard output are commonly known as filters. You may already be familiar with one of these commands, which the first one that I'm going to talk about.</p>

Regular expressions</h2>
A regular expression, or regex for short, is a string of text, that define a search pattern for a larger set of text. Regexes are used in many programs, such as in text editors, and search engines, and can be also of great use in the terminal</p>

Post scriptum</h2>
Before ending it for good, I want to show some other programs that might be of use in the Bash command line.</p>

File manipulation</h2>
I didn't get into detail in the previous part of this guide on the file manipulation side of things. However it is a crucial part of any computer system, after all they wouldn't be as useful as they are if you couldn't create, edit and delete files.</p>