Skip to main content

aws cli AuthFailure gotcha

It's probably been about 5-6 years since the last time I've used the AWS command line tools. The other day I signed up for the AWS Free Tier to familiarize myself with the aws command-line tools once again. I created myself a test user and granted full access to my aws account. Easy and simple. Well not so fast, initially my user account failed to authenticate properly. I tried re-generating new API keys and even created other different accounts with different types of access, but the problem persisted.

Problem: I was seeing the following:

[[email protected] ~]# aws ec2 describe-regions

An error occurred (AuthFailure) when calling the DescribeRegions operation: AWS was not able to validate the provided access credentials

I verified my AWS API credentials stored on ~/.aws/credentials to ensure everything was correct. Yet, the fucking problem persisted.

It turned out that the time on the system I was trying to use the aws cli tools in, had the time completely wrong! Once the time was fixed I was able to authenticate my account and use the aws cli tool.


Awesome Applications: 

Using Python to Output Pretty Looking JSON

Every once and awhile, theirs occasions that I have a giant glob of JSON data that I want to easily read it's data. Normally I opted to use The problem is whenever I use, I always have to be sure the JSON data doesn't include anything confidential. I was happy to learn that you can easily use the json library in Python to accomplish essentially the same thing.

For example:

alpha03:~ tony$ cat test.json
{"employees":[{"firstName":"John", "lastName":"Doe"},{"firstName":"Anna", "lastName":"Smith"},{"firstName":"Peter", "lastName":"Jones"}]} 
alpha03:~ tony$ python -m json.tool < test.json
    "employees": [
            "firstName": "John",
            "lastName": "Doe"
            "firstName": "Anna",
            "lastName": "Smith"
            "firstName": "Peter",
            "lastName": "Jones"


Hide Server and Web Application Information

First and foremost, this is security over obscurity. No relevant security improvement is gained by this, but rather just annoy or hopefully prevent automated bots or script kiddies from doing any future damage. By NO means it should be the only defensive mechanism for any site!

This site runs on Drupal. Drupal, by default returns Drupal specific http headers on every request. So I wanted to disabled this completely at the server level, in addition to the PHP and Apache information that is part of a typical stock LAMP configuration.

First, we start with PHP. The PHP X-Powered-By header can be disabled by disabling the expose_php option:

expose_php = Off

Next, is updating the default Server header set by Apache:

ServerTokens Prod

Finally, it's time to remove the X-Generator and X-Drupal-Cache specific Drupal headers.
Using Apache via mod_headers module:

<IfModule mod_headers.c>
     Header unset X-Generator
     Header unset X-Drupal-Cache

Using Nginx via the headers more module:

more_clear_headers 'x-generator';
more_clear_headers 'x-drupal-cache';

Why stop here when I can set custom headers. So as a joke, I want to tell the world that my sites are powered by Unicors and I’m the hacker being it. Doing so is dead simple.


add_header              X-Powered-By "Unicorns";
add_header              X-hacker "Alpha01";

Varnish (vcl_deliver)

set resp.http.X-Powered-By = "Unicorns";
set resp.http.X-hacker = "Alpha01";

Now, let's view my new http headers

alpha03:~ tony$ curl -I
HTTP/1.1 200 OK
Date: Thu, 24 Nov 2016 03:53:40 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Set-Cookie: __cfduid=d8fbf8e2a27fac74e782224db3fd3c86c1479959620; expires=Fri, 24-Nov-17 03:53:40 GMT; path=/;; HttpOnly
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: public, max-age=300
X-Content-Type-Options: nosniff
Content-Language: en
X-Frame-Options: SAMEORIGIN
Last-Modified: Tue, 22 Nov 2016 06:55:27 GMT
Vary: Cookie,Accept-Encoding
Front-End-Https: on
X-Powered-By: Unicorns
X-hacker: Alpha01
Server: cloudflare-nginx
CF-RAY: 3069ea499a6320ba-LAX



Awesome Applications: 

Compiling Nginx dynamically loadable modules

I’ve compiled plenty of dynamic loadable modules for Apache, but never for Nginx up until now. I must say compiling modules for Apache is a much easier process compared to Nginx.

I use Nginx as a SSL reverse proxy/terminator. I wanted to be able to remove certain http headers at the Nginx level rather than on my Apache backend. The way to achieve this is using a module called headers more , which is not part of core Nginx.

My Setup:
I’m using the nginx package installed via by the epel repository for CentOS 6.x. So compiling the module was a bit tricky initially.

Compilation Process:
1). Download the source code for the module, from

2). You need to find out the running version of Nginx:

$ nginx -v
nginx version: nginx/1.10.1

3). Download and extract the source code for that version of Nginx:

$ wget ''

4). Compilation:
On my first initial compilation, I mistakenly compiled the module using the following:

./configure --prefix=/opt/nginx \

Even though the module compiled successfully, when trying to load the module, it gave me the following error:

$ nginx -t
nginx: [emerg] module "/usr/lib64/nginx/modules/" is not binary compatible in /etc/nginx/nginx.conf:9
nginx: configuration file /etc/nginx/nginx.conf test failed

It turns out the module has to be compiled using the same ./configure flags as my running version of Nginx, in addition to the --add-dynamic-module=/opt/headers-more-nginx-module/headers-more-nginx-module-0.32 flag. Luckily, this can easily been obtained by running the following:

$ nginx -V
nginx version: nginx/1.10.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/ --lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=' -Wl,-E'

This is the tricky and annoying part. Since, the epel repo packaged binary of nginx was compiled with support of additional modules, the identical manual compilation that I was trying to do required some dependencies to be installed. In my case I just had to installed the following dependencies:

# yum install libxslt-devel perl-ExtUtils-Embed GeoIP-devel

After compilation completes. Then it's just a matter of adding the module to the nginx.conf config. In my case, I manually copied over the compiled module to /usr/lib64/nginx/modules, so my config looks like this:

load_module "/usr/lib64/nginx/modules/";

It is worth mentioning that Nginx is very picky on the order in which its configuration is set. For example. load_module option has be set before the events configuration block like this:

load_module "/usr/lib64/nginx/modules/";

events {
    worker_connections  1024;

Otherwise, you'll get the following error when trying to load the module.

# nginx -t
nginx: [emerg] "load_module" directive is specified too late in /etc/nginx/nginx.conf:14
nginx: configuration file /etc/nginx/nginx.conf test failed

One really important thing to keep in mind, is that you'll need to recompile the module each time Nginx gets updated.


Awesome Applications: 

Redirect to www domain in Varnish

I’m a domain name hoarder, over twenty domain names and counting. I’ve done plenty of 301 domain name redirects in the past at the Apache level using mod_rewrite. Hell, just about every registrar now in days gives you the ability to this really easily through them.

In my case, I opted to simply point DNS directly to my server. Since I’m using Varnish (3.x), it’s more efficient to do the actual domain name redirect in Varnish itself rather than Apache. Also not to mention, the actual configuration is easier to understand in Varnish than using mod_rewrite in my opinion.

Example Varnish 3.x config:

# Tony's shitty portfolio site
sub vcl_recv {
  if ( ~ "(?i)^(www.)?(tony|antonio)baltazar\.(com|org|net)") {
       set = "";
       error 750 "http://" + + req.url;

sub vcl_error {
  if (obj.status == 750) {
    set obj.http.Location = obj.response;
    set obj.status = 301;

On this example, any incoming request matching (?i)^(www.)?(tony|antonio)baltazar\.(com|org|net) will have the host request header set to The error function call with the status method of 750 is simply used internally by Varnish, so we can track it at the vcl_error subroutine. I can see why in Varnish 4.x this is handled differently since, this is basically a synthetic internal Varnish request that we're working on and should not be handled in vcl_error.

Example Varnish 4 config

sub vcl_recv {
  if ( ~ "(?i)^(www.)?(tony|antonio)baltazar\.(com|org|net)") {
    return (synth(750, ""));

sub vcl_synth {
    if (resp.status == 750) {
        set resp.status = 301;
        set resp.http.Location = "" + req.url;

So in Varnish 4, the configuration is even easier to understand.

An additional benefit of using this configuration is that now I’m consolidating and increasing Varnish cache hit rate. This means that Varnish won’t have two different caches for and

Awesome Applications: 

Python if __name__ == '__main__' Ruby equivalent

Python is by no chance my favorite language to work in, however I always loved the way you can beautifully write your modules and easily test them via if __name__ == '__main__' statement. I've been doing a lot of Ruby programing these past few weeks, and I came across a situation were I needed this exact feature in Ruby.

My problem:
I needed to run some unit tests to a Ruby based TCP server that gets spawn as daemon. The program is completely command line, and once spawned, their isn't any code to communicate with its child process. The unit tests itself aren't exactly to complicated. I simply need to make sure that the TCP server starts, verify the status of it using its PID, and be able to kill the process. I needed to run the unit tests without heavily modifying the existing program, and the best best to accomplished it was using a similar if __name__ == '__main__' Python approach. Lucky for me, in the Ruby world we can accomplish the awesome if __name__ == '__main__' Python statement via if __FILE__ == $0

Here is a test module called test-module.rb

#!/usr/bin/env ruby

if __FILE__ == $0
  puts "Executed via command line."
  puts "Included."

Now, if we run this test-module.rb from the command line, the if __FILE__ == $0 block will evaluate to true.

alpha03:tests $ ./test-module.rb
Executed via command line.

If the module gets included the if __FILE__ == $0 block will evaluate to false. Example script called test.rb

#!/usr/bin/env ruby

require './test-module'

Running the test.rb script that required test-module.rb

alpha03:tests tony$ ./test.rb

Ruby rocks!


Time Stamping Bash's command history

It seems sharing your dot (.) config files is an act that all cool kids do these days. I won't be sharing my Bash configs, however I will share one cool Bash shell trick of time stamping your command history, I use in all of my systems and servers. This is accomplished using the HISTTIMEFORMAT environment variable. Using standard date format control ouput syntax, it's fairly easy to customize the command history time stamp to whatever time format output you prefer. Finally, in addition to using a customized HISTTIMEFORMAT value, I also add the HISTSIZE environment variable. This environment variable lets you override the default command history count to a much larger history count.


Sample output:

3318 21/04/2016 04:42:20 ls
3318 21/04/2016 04:43:29 vim .bashrc
3319 21/04/2016 04:43:37 vim .bash_profile
3320 21/04/2016 05:03:58 man date
3321 21/04/2016 05:07:43 history |tail
3322 21/04/2016 05:07:47 history |tail -5


Best scripting/programming language to learn from an Ops perspective

Knowing some sort of programming is an absolute must for any systems administrator. Hell, even in the Microsoft world, the necessity can't be anymore apparent with the creation of PowerShell. So what is the best language to learn for a beginner? This is a question that many junior sysadmins or students asked themselves when starting of.

Having been a systems administrator for well over eight years now, and worked with Bash, Perl, PHP, Python, and Ruby extensively within those eight years. I think this is a two part answer. In my opinion the best language to start is Bash/shell. Bash/shell is everywhere. Having an advanced Bash knowledge automatically makes you a better sysadmin. This is because Bash/shell scripting is practically glued to other external *nix userland tools (grep, sort, tail, awk, sed, etc..). Knowing Bash extensively, makes you a better command line user. The second of the answer to which is the best scripting/programming language to learn, depends on what environment you work in, or want to work in. If you work or want to work in a Ruby on Rails shop, then obviously you would want to opt learn Ruby, likewise if you work in a WordPress shop, you should opt for learning PHP.

Programming vs Scripting
A systems administrator is NOT a software engineer. There is a huge difference between scripting some sort of convoluted process (automation) and in architecting a complex piece of software. So it is important to acknowledge that the skill set between the both is not the same. Most of the religious wars between a programming language preference people love to display on the internet, is mainly from a developers perspective. Whether the arguments are valid or not, in my opinion they are irrelevant from a sysadmin's point of view. Basically we must know how to use an API and not necessarily how to create one. While in fact it helps to know how to construct your objects by knowing how to create elegant classes, methods, and attributes. It's not vital for a sysadmin to know the exact details on how the API/library was created.

Programming Experience
Once you know a scripting language reasonably well. By knowing either Ruby, Perl, Python, or PHP, you practically can't avoid being already familiar with any of the other scripting languages mentioned. These languages are so similar, it's fairly easy to pick up book and learn another scripting language in a short time span.

It's no secret Ruby is my favorite language, however I'm not overly religious about it. It's not the end of the world if I don't get to use Ruby everyday. In fact, if I need to write a script really fast. Usually my first choices are either Bash or Perl. This is mainly because these are the languages I've known the longest, and for reason I'm able to program in much faster. For me it all boils down to which language will take me less time hack around with and which tool is best for the job. This usually means, is there an existing program, API/library out there that does much of the work I need to solve my problem.

Thus said, when I first started my career, Bash and Perl the defacto scripting languages for a *nix admin to know, now in days it seems Python and Ruby are what companies prefer. Even now with the popularity of NodeJS, soon JavaScript will come into the front along with the other traditional major scripting languages.

Happy hacking.

Packt Publishing Free E-Books crawler

I'm a big fan of Packt Publishing, and have purchased quite a few books from them. So when I first heard a couple of months back that they were going to give out free e-books everyday, my jaw literally dropped.

I've normally been manually checking the site everyday for books that I might be interested on reading. The problem with this, is that their have been days that I missed out getting some free books that I would've love to read. So I wrote a short script that would notify me if there's a free book available that I might be interested in reading. I would've love if Packt Publishing provided an rss feed so I can easily get notifications of their free books. Thus said, I really can't complain since they're already kind enough to give the world free e-books to spread knowledge.


Locking Down WordPress Access with Varnish 3.x

I have Varnish in front of all my WordPress sites and configured all /wp-admin traffic use https via Nginx. See

So to lock down access to my WordPress site's requires both Varnish and Nginx configs to be modified.

Block at the http Varnish level:

sub vcl_recv {
    if ((req.url ~ "wp-(login|admin)") && (client.ip !~ MY-IP-ADDRESS)) {
                error 403 "Fuck off";

Block at the https Nginx level (using as an example):

                location /wp-admin {
                        allow   MY-IP-ADDRESS;
                        deny all;
                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                        proxy_set_header Host $http_host;
                location /wp-login.php {
                        allow MY-IP-ADDRESS;
                        deny all;
                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                        proxy_set_header Host $http_host;


Awesome Applications: 


Premium Drupal Themes by Adaptivethemes