Tag Cloud
Currently Reading
Latest Book Reviews
- Certified Kubernetes Application Developer (CKAD) Study Guide, 2nd Edition Posted on January 11, 2025
- Rancher Deep Dive Posted on March 31, 2023
- Leveraging Kustomize for Kubernetes Manifests Posted on March 24, 2023
- Automating Workflows with GitHub Actions Posted on October 13, 2022
- Deep-Dive Terraform on Azure Posted on August 30, 2022 All Book Reviews
Latest Posts
- Compiling Nginx dynamically loadable modules Posted on November 18, 2016
- Redirect to www domain in Varnish Posted on July 31, 2016
- Python if __name__ == '__main__' Ruby equivalent Posted on July 12, 2016
- Time Stamping Bash's command history Posted on April 21, 2016
- Best scripting/programming language to learn from an Ops perspective Posted on April 20, 2016
November 18, 2016
Compiling Nginx dynamically loadable modules
by Alpha01
I’ve compiled plenty of dynamic loadable modules for Apache, but never for Nginx up until now. I must say compiling modules for Apache is a much easier process compared to Nginx.
I use Nginx as a SSL reverse proxy/terminator. I wanted to be able to remove certain http headers at the Nginx level rather than on my Apache backend. The way to achieve this is using a module called headers more, which is not part of core Nginx.
My Setup
I’m using the nginx package installed via by the epel repository for CentOS 6.x. So compiling the module was a bit tricky initially.
Compilation Process
1). Download the source code for the module, from https://github.com/openresty/headers-more-nginx-module/tags
2). You need to find out the running version of Nginx:
$ nginx -v
nginx version: nginx/1.10.1
3). Download and extract the source code for that version of Nginx:
wget 'http://nginx.org/download/nginx-1.10.1.tar.gz'
4). Compilation: On my first initial compilation, I mistakenly compiled the module using the following:
./configure --prefix=/opt/nginx \
--add-dynamic-module=/opt/headers-more-nginx-module/headers-more-nginx-module-0.32
Even though the module compiled successfully, when trying to load the module, it gave me the following error:
nginx -t
nginx: [emerg] module "/usr/lib64/nginx/modules/ngx_http_headers_more_filter_module.so" is not binary compatible in /etc/nginx/nginx.conf:9
nginx: configuration file /etc/nginx/nginx.conf test failed
It turns out the module has to be compiled using the same ./configure flags as my running version of Nginx, in addition to the --add-dynamic-module=/opt/headers-more-nginx-module/headers-more-nginx-module-0.32 flag. Luckily, this can easily been obtained by running the following:
nginx -V
nginx version: nginx/1.10.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=' -Wl,-E'
This is the tricky and annoying part. Since, the epel repo packaged binary of nginx was compiled with support of additional modules, the identical manual compilation that I was trying to do required some dependencies to be installed. In my case I just had to installed the following dependencies:
yum install libxslt-devel perl-ExtUtils-Embed GeoIP-devel
After compilation completes. Then it’s just a matter of adding the module to the nginx.conf config. In my case, I manually copied over the compiled module to /usr/lib64/nginx/modules, so my config looks like this:
load_module "/usr/lib64/nginx/modules/ngx_http_headers_more_filter_module.so";
It is worth mentioning that Nginx is very picky on the order in which its configuration is set. For example. load_module option has be set before the events configuration block like this:
load_module "/usr/lib64/nginx/modules/ngx_http_headers_more_filter_module.so";
events {
worker_connections 1024;
}
Otherwise, you’ll get the following error when trying to load the module.
nginx -t
nginx: [emerg] "load_module" directive is specified too late in /etc/nginx/nginx.conf:14
nginx: configuration file /etc/nginx/nginx.conf test failed
Important
One really important thing to keep in mind, is that you’ll need to recompile the module each time Nginx gets updated!
Tags: [centos nginx July 31, 2016
Redirect to www domain in Varnish
by Alpha01
I’m a domain name hoarder, over twenty domain names and counting. I’ve done plenty of 301 domain name redirects in the past at the Apache level using mod_rewrite. Hell, just about every registrar now in days gives you the ability to this really easily through them.
In my case, I opted to simply point DNS directly to my server. Since I’m using Varnish (3.x), it’s more efficient to do the actual domain name redirect in Varnish itself rather than Apache. Also not to mention, the actual configuration is easier to understand in Varnish than using mod_rewrite in my opinion.
Example Varnish 3.x config:
# Tony's shitty portfolio site
sub vcl_recv {
if (req.http.host ~ "(?i)^(www.)?(tony|antonio)baltazar\.(com|org|net)") {
set req.http.host = "www.antoniobaltazar.com";
error 750 "http://" + req.http.host + req.url;
}
}
sub vcl_error {
if (obj.status == 750) {
set obj.http.Location = obj.response;
set obj.status = 301;
return(deliver);
}
}
On this example, any incoming request matching (?i)^(www.)?(tony|antonio)baltazar\.(com|org|net) will have the host request header set to www.antoniobaltazar.com. The error function call with the status method of 750 is simply used internally by Varnish, so we can track it at the vcl_error subroutine. I can see why in Varnish 4.x this is handled differently since, this is basically a synthetic internal Varnish request that we’re working on and should not be handled in vcl_error.
Example Varnish 4 config:
sub vcl_recv {
if (req.http.host ~ "(?i)^(www.)?(tony|antonio)baltazar\.(com|org|net)") {
return (synth(750, ""));
}
}
sub vcl_synth {
if (resp.status == 750) {
set resp.status = 301;
set resp.http.Location = "http://www.antoniobaltazar.com" + req.url;
return(deliver);
}
}
So in Varnish 4, the configuration is even easier to understand.
An additional benefit of using this configuration is that now I’m consolidating and increasing Varnish cache hit rate. This means that Varnish won’t have two different caches for antoniobaltazar.com and www.antoniobaltazar.com.
Tags: [varnish July 12, 2016
Python if __name__ == '__main__' Ruby equivalent
by Alpha01
Python is by no chance my favorite language to work in, however I always loved the way you can beautifully write your modules and easily test them via if __name__ == '__main__' statement. I’ve been doing a lot of Ruby programing these past few weeks, and I came across a situation were I needed this exact feature in Ruby.
My problem
I needed to run some unit tests to a Ruby based TCP server that gets spawn as daemon. The program is completely command line, and once spawned, their isn’t any code to communicate with its child process. The unit tests itself aren’t exactly to complicated. I simply need to make sure that the TCP server starts, verify the status of it using its PID, and be able to kill the process. I needed to run the unit tests without heavily modifying the existing program, and the best best to accomplished it was using a similar if __name__ == '__main__' Python approach. Lucky for me, in the Ruby world we can accomplish the awesome if __name__ == '__main__' Python statement via if __FILE__ == $0.
Example
Here is a test module called test-module.rb
#!/usr/bin/env ruby
if __FILE__ == $0
puts "Executed via command line."
else
puts "Included."
end
Now, if we run this test-module.rb from the command line, the if __FILE__ == $0 block will evaluate to true.
alpha03:tests $ ./test-module.rb
Executed via command line.
If the module gets included the if __FILE__ == $0 block will evaluate to false. Example script called test.rb
#!/usr/bin/env ruby
require './test-module'
Running the test.rb script that required test-module.rb
alpha03:tests tony$ ./test.rb
Included.
Conclusion
Ruby rocks!
Tags: [ruby python April 21, 2016
Time Stamping Bash's command history
by Alpha01
It seems sharing your dot (.) config files is an act that all cool kids do these days. I won’t be sharing my Bash configs, however I will share one cool Bash shell trick of time stamping your command history, I use in all of my systems and servers. This is accomplished using the HISTTIMEFORMAT environment variable. Using standard date format control output syntax, it’s fairly easy to customize the command history time stamp to whatever time format output you prefer. Finally, in addition to using a customized HISTTIMEFORMAT value, I also add the HISTSIZE environment variable. This environment variable lets you override the default command history count to a much larger history count.
HISTSIZE=10000
HISTTIMEFORMAT="%d/%m/%Y %T "
export HISTTIMEFORMAT
Usage
Sample output:
3318 21/04/2016 04:42:20 ls
3318 21/04/2016 04:43:29 vim .bashrc
3319 21/04/2016 04:43:37 vim .bash_profile
3320 21/04/2016 05:03:58 man date
3321 21/04/2016 05:07:43 history |tail
3322 21/04/2016 05:07:47 history |tail -5
bash April 20, 2016
Best scripting/programming language to learn from an Ops perspective
by Alpha01
Background
Knowing some sort of programming is an absolute must for any systems administrator. Hell, even in the Microsoft world, the necessity can’t be anymore apparent with the creation of PowerShell. So what is the best language to learn for a beginner? This is a question that many junior sysadmins or students asked themselves when starting of.
Having been a systems administrator for well over eight years now, and worked with Bash, Perl, PHP, Python, and Ruby extensively within those eight years. I think this is a two part answer. In my opinion the best language to start is Bash/shell. Bash/shell is everywhere. Having an advanced Bash knowledge automatically makes you a better sysadmin. This is because Bash/shell scripting is practically glued to other external *nix userland tools (grep, sort, tail, awk, sed, etc..). Knowing Bash extensively, makes you a better command line user. The second of the answer to which is the best scripting/programming language to learn, depends on what environment you work in, or want to work in. If you work or want to work in a Ruby on Rails shop, then obviously you would want to opt learn Ruby, likewise if you work in a WordPress shop, you should opt for learning PHP.
Programming vs Scripting
A systems administrator is NOT a software engineer. There is a huge difference between scripting some sort of convoluted process (automation) and in architecting a complex piece of software. So it is important to acknowledge that the skill set between the both is not the same. Most of the religious wars between a programming language preference people love to display on the internet, is mainly from a developers perspective. Whether the arguments are valid or not, in my opinion they are irrelevant from a sysadmin’s point of view. Basically we must know how to use an API and not necessarily how to create one. While in fact it helps to know how to construct your objects by knowing how to create elegant classes, methods, and attributes. It’s not vital for a sysadmin to know the exact details on how the API/library was created.
Programming Experience
Once you know a scripting language reasonably well. By knowing either Ruby, Perl, Python, or PHP, you practically can’t avoid being already familiar with any of the other scripting languages mentioned. These languages are so similar, it’s fairly easy to pick up book and learn another scripting language in a short time span.
It’s no secret Ruby is my favorite language, however I’m not overly religious about it. It’s not the end of the world if I don’t get to use Ruby everyday. In fact, if I need to write a script really fast. Usually my first choices are either Bash or Perl. This is mainly because these are the languages I’ve known the longest, and for reason I’m able to program in much faster. For me it all boils down to which language will take me less time hack around with and which tool is best for the job. This usually means, is there an existing program, API/library out there that does much of the work I need to solve my problem.
Thus said, when I first started my career, Bash and Perl the defacto scripting languages for a *nix admin to know, now in days it seems Python and Ruby are what companies prefer. Even now with the popularity of NodeJS, soon JavaScript will come into the front along with the other traditional major scripting languages.
Happy hacking!
Tags: [php perl go python bash ruby