Tag Cloud
Currently Reading
Latest Book Reviews
- Rancher Deep Dive Posted on March 31, 2023
- Leveraging Kustomize for Kubernetes Manifests Posted on March 24, 2023
- Automating Workflows with GitHub Actions Posted on October 13, 2022
- Deep-Dive Terraform on Azure Posted on August 30, 2022
- Effective DevOps Posted on January 5, 2022 All Book Reviews
Latest Posts
- aws cli AuthFailure gotcha Posted on January 30, 2017
- Using Python to Output Pretty Looking JSON Posted on December 18, 2016
- Hide Server and Web Application Information Posted on November 22, 2016
- Compiling Nginx dynamically loadable modules Posted on November 18, 2016
- Redirect to www domain in Varnish Posted on July 31, 2016
January 30, 2017
aws cli AuthFailure gotcha
by Alpha01
It’s probably been about 5-6 years since the last time I’ve used the AWS command line tools. The other day I signed up for the AWS Free Tier to familiarize myself with the aws command-line tools once again. I created myself a test user and granted full access to my aws account. Easy and simple. Well not so fast, initially my user account failed to authenticate properly. I tried re-generating new API keys and even created other different accounts with different types of access, but the problem persisted.
Problem
I was getting the following error:
[root@bashninja ~]# aws ec2 describe-regions
An error occurred (AuthFailure) when calling the DescribeRegions operation: AWS was not able to validate the provided access credentials
I verified my AWS API credentials stored on ~/.aws/credentials to ensure everything was correct. Yet, the fucking problem persisted.
Fix
It turned out that the time on the system I was trying to use the aws cli tools in, had the time completely wrong! Once the time was fixed I was able to authenticate my account and use the aws cli tool.
Tags: [aws python December 18, 2016
Using Python to Output Pretty Looking JSON
by Alpha01
Every once and awhile, theirs occasions that I have a giant glob of JSON data that I want to easily read it’s data. Normally I opted to use http://jsonlint.com/. The problem is whenever I use http://jsonlint.com/, I always have to be sure the JSON data doesn’t include anything confidential. I was happy to learn that you can easily use the json library in Python to accomplish essentially the same thing.
For example:
alpha03:~ tony$ cat test.json
{"employees":[{"firstName":"John", "lastName":"Doe"},{"firstName":"Anna", "lastName":"Smith"},{"firstName":"Peter", "lastName":"Jones"}]}
alpha03:~ tony$ python -m json.tool < test.json
{
"employees": [
{
"firstName": "John",
"lastName": "Doe"
},
{
"firstName": "Anna",
"lastName": "Smith"
},
{
"firstName": "Peter",
"lastName": "Jones"
}
]
}
python November 22, 2016
Hide Server and Web Application Information
by Alpha01
First and foremost, this is security over obscurity. No relevant security improvement is gained by this, but rather just annoy or hopefully prevent automated bots or script kiddies from doing any future damage. By NO means it should be the only defensive mechanism for any site!
This site runs on Drupal. Drupal, by default returns Drupal specific http headers on every request. So I wanted to disabled this completely at the server level, in addition to the PHP and Apache information that is part of a typical stock LAMP configuration.
First, we start with PHP. The PHP X-Powered-By header can be disabled by disabling the expose_php option:
expose_php = Off
Next, is updating the default Server header set by Apache:
ServerTokens Prod
Finally, it’s time to remove the X-Generator and X-Drupal-Cache specific Drupal headers. Using Apache via mod_headers module:
<IfModule mod_headers.c>
Header unset X-Generator
Header unset X-Drupal-Cache
</IfModule>
Using Nginx via the headers more module:
more_clear_headers 'x-generator';
more_clear_headers 'x-drupal-cache';
Why stop here when I can set custom headers. So as a joke, I want to tell the world that my sites are powered by Unicors and I’m the hacker being it. Doing so is dead simple.
- Nginx
add_header X-Powered-By "Unicorns";
add_header X-hacker "Alpha01";
- Varnish (vcl_deliver)
set resp.http.X-Powered-By = "Unicorns";
set resp.http.X-hacker = "Alpha01";
View our changes
Now, let’s view my new http headers
alpha03:~ tony$ curl -I https://www.rubysecurity.org
HTTP/1.1 200 OK
Date: Thu, 24 Nov 2016 03:53:40 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Set-Cookie: __cfduid=d8fbf8e2a27fac74e782224db3fd3c86c1479959620; expires=Fri, 24-Nov-17 03:53:40 GMT; path=/; domain=.rubysecurity.org; HttpOnly
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: public, max-age=300
X-Content-Type-Options: nosniff
Content-Language: en
X-Frame-Options: SAMEORIGIN
Last-Modified: Tue, 22 Nov 2016 06:55:27 GMT
Vary: Cookie,Accept-Encoding
Front-End-Https: on
X-Powered-By: Unicorns
X-hacker: Alpha01
Server: cloudflare-nginx
CF-RAY: 3069ea499a6320ba-LAX
Resources
- http://php.net/manual/en/ini.core.php#ini.expose-php
- http://httpd.apache.org/docs/2.4/mod/core.html#servertokens
- https://www.drupal.org/node/982034#comment-4719282
- http://nginx.org/en/docs/http/ngx_http_headers_module.html
php drupal nginx varnish apache security November 18, 2016
Compiling Nginx dynamically loadable modules
by Alpha01
I’ve compiled plenty of dynamic loadable modules for Apache, but never for Nginx up until now. I must say compiling modules for Apache is a much easier process compared to Nginx.
I use Nginx as a SSL reverse proxy/terminator. I wanted to be able to remove certain http headers at the Nginx level rather than on my Apache backend. The way to achieve this is using a module called headers more, which is not part of core Nginx.
My Setup
I’m using the nginx package installed via by the epel repository for CentOS 6.x. So compiling the module was a bit tricky initially.
Compilation Process
1). Download the source code for the module, from https://github.com/openresty/headers-more-nginx-module/tags
2). You need to find out the running version of Nginx:
$ nginx -v
nginx version: nginx/1.10.1
3). Download and extract the source code for that version of Nginx:
wget 'http://nginx.org/download/nginx-1.10.1.tar.gz'
4). Compilation: On my first initial compilation, I mistakenly compiled the module using the following:
./configure --prefix=/opt/nginx \
--add-dynamic-module=/opt/headers-more-nginx-module/headers-more-nginx-module-0.32
Even though the module compiled successfully, when trying to load the module, it gave me the following error:
nginx -t
nginx: [emerg] module "/usr/lib64/nginx/modules/ngx_http_headers_more_filter_module.so" is not binary compatible in /etc/nginx/nginx.conf:9
nginx: configuration file /etc/nginx/nginx.conf test failed
It turns out the module has to be compiled using the same ./configure flags as my running version of Nginx, in addition to the --add-dynamic-module=/opt/headers-more-nginx-module/headers-more-nginx-module-0.32 flag. Luckily, this can easily been obtained by running the following:
nginx -V
nginx version: nginx/1.10.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=' -Wl,-E'
This is the tricky and annoying part. Since, the epel repo packaged binary of nginx was compiled with support of additional modules, the identical manual compilation that I was trying to do required some dependencies to be installed. In my case I just had to installed the following dependencies:
yum install libxslt-devel perl-ExtUtils-Embed GeoIP-devel
After compilation completes. Then it’s just a matter of adding the module to the nginx.conf config. In my case, I manually copied over the compiled module to /usr/lib64/nginx/modules, so my config looks like this:
load_module "/usr/lib64/nginx/modules/ngx_http_headers_more_filter_module.so";
It is worth mentioning that Nginx is very picky on the order in which its configuration is set. For example. load_module option has be set before the events configuration block like this:
load_module "/usr/lib64/nginx/modules/ngx_http_headers_more_filter_module.so";
events {
worker_connections 1024;
}
Otherwise, you’ll get the following error when trying to load the module.
nginx -t
nginx: [emerg] "load_module" directive is specified too late in /etc/nginx/nginx.conf:14
nginx: configuration file /etc/nginx/nginx.conf test failed
Important
One really important thing to keep in mind, is that you’ll need to recompile the module each time Nginx gets updated!
Tags: [centos nginx July 31, 2016
Redirect to www domain in Varnish
by Alpha01
I’m a domain name hoarder, over twenty domain names and counting. I’ve done plenty of 301 domain name redirects in the past at the Apache level using mod_rewrite. Hell, just about every registrar now in days gives you the ability to this really easily through them.
In my case, I opted to simply point DNS directly to my server. Since I’m using Varnish (3.x), it’s more efficient to do the actual domain name redirect in Varnish itself rather than Apache. Also not to mention, the actual configuration is easier to understand in Varnish than using mod_rewrite in my opinion.
Example Varnish 3.x config:
# Tony's shitty portfolio site
sub vcl_recv {
if (req.http.host ~ "(?i)^(www.)?(tony|antonio)baltazar\.(com|org|net)") {
set req.http.host = "www.antoniobaltazar.com";
error 750 "http://" + req.http.host + req.url;
}
}
sub vcl_error {
if (obj.status == 750) {
set obj.http.Location = obj.response;
set obj.status = 301;
return(deliver);
}
}
On this example, any incoming request matching (?i)^(www.)?(tony|antonio)baltazar\.(com|org|net) will have the host request header set to www.antoniobaltazar.com. The error function call with the status method of 750 is simply used internally by Varnish, so we can track it at the vcl_error subroutine. I can see why in Varnish 4.x this is handled differently since, this is basically a synthetic internal Varnish request that we’re working on and should not be handled in vcl_error.
Example Varnish 4 config:
sub vcl_recv {
if (req.http.host ~ "(?i)^(www.)?(tony|antonio)baltazar\.(com|org|net)") {
return (synth(750, ""));
}
}
sub vcl_synth {
if (resp.status == 750) {
set resp.status = 301;
set resp.http.Location = "http://www.antoniobaltazar.com" + req.url;
return(deliver);
}
}
So in Varnish 4, the configuration is even easier to understand.
An additional benefit of using this configuration is that now I’m consolidating and increasing Varnish cache hit rate. This means that Varnish won’t have two different caches for antoniobaltazar.com and www.antoniobaltazar.com.
Tags: [varnish