Docker Postgres “PANIC: could not locate a valid checkpoint record”

It seemed that my Postgres database was not properly shut down when rebooting and when I tried to use docker-compose to start it again, the following message was shown in docker logs:

To fix this, first shut down this container (docker-compose down), then start the container in interactive mode:

After the transaction log is reset, everything should be fine. Now you can start your containers again (docker-compose up -d).

Fixing iPhone 6S WiFi issue

My iPhone very often has unstable connections. It’s connected to the WiFi but has no internet access. I’ve tried resetting the network, rebooting, and forgetting the network but it doesn’t work. Then I found there seems to be an easier option: just enter wifi settings and hit the blue i icon and then you’ll see the following interface. Simply hit the renew lease button on the bottom and voilà! You’re online again!

Exporting and Importing Elasticsearch Indicies

In my project I need to run some local tests with data from a production elasticsearch cluster, so I exported data from the production server and imported to my local cluster. This can also be used when backing up and restoring data. Here’re the instructions.

Before you start, check out the official documentation: Snapshot and Restore.

Backing up/exporting data:

  1. Modify your eleasticsearch configuration file (normally elasticsearch.yml) and add a path.repo line, for example:
  2. Make sure this path has the correct permissions so that elasticsearch can read and write.
  3. Create snapshot:
  4. Copy the files in the configured location to your local machine.

Restoring/importing data:

  1. Modify your local elasticsearch configuration similarly like step 1 when backing up.
  2. Place the snapshot files to the repo path.
  3. Close your indices:
  4. Import data:
  5. Reopen your indices:

It is important that your the elasticsearch version on your importing party is compatible with the one exporting data, i.e., in this case your local machine has to be the same version or newer. If not, you need to upgrade elasticsearch first. The official documentation says:

The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore it’s possible to restore a snapshot made from one cluster into another cluster. All that is required is registering the repository containing the snapshot in the new cluster and starting the restore process. The new cluster doesn’t have to have the same size or topology. However, the version of the new cluster should be the same or newer than the cluster that was used to create the snapshot.

Installing Theano and CUDA on Mac OS X

I started trying Theano today and wanted to use the GPU (NVIDIA GeForce GT 750M 2048 MB) on my Mac. Here’s a brief instruction on how to use the GPU on Mac, largely following the instructions from http://deeplearning.net/software/theano/install.html#mac-os.

Install Theano:

Download and install CUDA: https://developer.nvidia.com/cuda-downloads

Put the following lines into your ~/.bash_profile:

Note that the PATH line is necessary. Otherwise you may see the following message:

ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.

Configure Theano:

Test if GPU is used:

A more realistic example:

So it seems this GPU does not outperform the CPU. Well,GT 750M may not be the best GPU you can get… Someone else here has a similar experience.

 

Migrating/Installing Prestashop to Amazon AWS EC2

Recently I migrated a Prestashop website to AWS/EC2 and the process is quite smooth. Here’s how to do it.

1. Get your Website data: including Prestashop website files and a latest database dump. Remove all the files except index.php in /cache/smarty/compile and /cache/smarty/cache. (See here.)

2. Launch your EC2 instance. But before doing that, check first if a pre-configured or managed Prestashop is available in your preferred region (more info here). If not, choose the appropriate instance type and get it running. In my case I chose Ubuntu Linux.

3. Install the AMP (Apache, MySQL and PHP). On Ubuntu you can simply do it with a few simple commands:

Make sure that you remember your password for MySQL root user and phpmyadmin user.

4. Copy your database dump and website files to your EC2 instance using scp, for instance:

5. Create a MySQL user for Prestashop, you don’t want to use the root user for this purpose. You can do this using phpmyadmin or a command line. Also, create a database for Prestashop and grant the user you’ve created just now all privileges on this database.

6. Import your database from the dump:

7. Extract your website files on your EC2 instance. In my case, inside /var/www/html/. If there is an index.html file inside this directory, remove it first. Make sure you set the correct user and group to all the website files:

Note that if don’t set the correct permissions, you may get complaints from Apache, like the following:

PHP Fatal error:  Uncaught exception ‘SmartyException’ with message ‘unable to write file /var/www/html/cache/smarty/compile/46/d9/69/wrt54ecd6c1856a72.61943173’ in /var/www/html/tools/smarty/sysplugins/smarty_internal_write_file.php:44

Stack trace:

#0 /var/www/html/tools/smarty/sysplugins/smarty_internal_template.php(201): Smarty_Internal_Write_File::writeFile(‘/var/www/html/c…’, ‘<?php /* Smarty…’, Object(Smarty))

#1 /var/www/html/tools/smarty/sysplugins/smarty_internal_templatebase.php(155): Smarty_Internal_Template->compileTemplateSource()

#2 /var/www/html/classes/controller/AdminController.php(1936): Smarty_Internal_TemplateBase->fetch(‘controllers/mod…’)

#3 /var/www/html/classes/controller/AdminController.php(2337): AdminControllerCore->initModal()

#4 /var/www/html/classes/controller/Controller.php(163): AdminControllerCore->init()

#5 /var/www/html/classes/Dispatcher.php(373): ControllerCore->run()

#6 /var/www/html/admin123/index.php(54): DispatcherCore->dispatch()

#7 {main}\n  thrown in /var/www/html/tools/smarty/sysplugins/smarty_internal_write_file.php on line 44

8. Enable mcrypt for PHP:

9. Update your Prestashop settings in the config/settings.inc.php file. Set the correct database information.

10. Enable mod_rewrite:

Also, make sure to add something like the following in your apache site configuration file:

And then remember to restart your apache server.

11. Install sendmail:

12. Go to your Prestashop back office and reconfigure your domain. Go to Preferences -> SEO & URLs and scroll down to Set shop URL section.

Now everything should be OK, your new Prestashop website is available on your new domain!

If you have any issues, just drop a line here or email me.

OSX Yosemite cannot boot when TRIM is enabled on third-party SSD

After upgrading my MBP to OSX Yosemite and enabling TRIM, the computer cannot boot into the system after restart. It stuck with a no-entry sign.

You can solve this issue by following instructions from http://www.cindori.org/trim-in-os-x-yosemite/

Boot your Mac in Recovery Mode by holding Cmd-R during boot

Open the Terminal from the menu bar

Run these commands, replacing YourDisk with the name of your Mac disk

rm -rf /Volumes/YourDisk/System/Library/Extensions/IOAHCIFamily.kext
cp -r /System/Library/Extensions/IOAHCIFamily.kext /Volumes/YourDisk/System/Library/Extensions/IOAHCIFamily.kext
touch /Volumes/YourDisk/System/Library/Extensions
kextcache -u /Volumes/YourDisk

For me, I have to first use the disk utility to unlock my SSD first since I have Filevault enabled, otherwise the first command above will complain about “readonly” files.

Since TRIM is very helpful for SSDs, it’s encouraged to have it enabled for better performance. You can use Trim Enabler v3.2.5 and above without issues. But note that it disables kext signing on your mac, “it still leaves you with the same amount of security as in OS X Mavericks, where the kext signing requirement didn’t exist”. (http://www.cindori.org/update-on-trim-in-yosemite/)

WordPress pages load very slowly after Google is blocked

After Google is completely blocked in China since May 31, I’ve had problems loading my website. After some investigation I found out the problem lies in loading failure of Google Fonts. Since it takes a long time to fail, the page waits before it fails and display with the fallback styles.

To solve the problem, I crawled the font styles from Google and put it up on my server. Then I changed the WordPress theme to use the font style on the server. Everything seems to work perfectly after that.

Another option would be completely disable Google Font Styles. This can be easily done using the WP plugin “Disable Google Fonts”, if you’re using WP’s default twenty-* themes. In my case, I would have to comment out two lines in Mog Theem Functions file (functions.php). Since my theme relies on Google Fonts, instead of commenting out the lines, I changed these to lines to use the localized style file:

 

Raspberry Pi as a toy web server

1. Tell my router to route certain traffic to the Raspberry Pi. For example, I route HTTP and SSH traffic to one of my Pi’s. I disabled password login for  SSH, using public key authentication instead: in /etc/ssh/sshd_config, use the following setting and restart SSH service using `sudo service ssh restart’.

2. Getting my IP address of the Raspberry Pi. I firstly created a PHP script on my domain to record the IP address in a text file.

Then tell the Raspberry Pi to report its IP address every 5 minutes, using crontab:

After a while the IP address is recorded in the text file and updated every 5 minutes.

3. Optionally create a DNS record for the Pi. I use Cloudflare to manage DNS settings by myself, so just add/update an entry in Cloudflare’s settings. I point pi.daoyuan.li to the IP address of one Pi. This can be done automatically in the future.

4. Install Flask on the Pi.

5. Install nginx and uwsgi on the Pi.

6. Set up nginx along with uwsgi and Flask.

Edit nginx config:

Edit uwsgi config:

Create a simple Flask app:

Start up uwsgi:

7. Done! http://pi.daoyuan.li:5000/

 Update on June 24, 2014:

Getting the external address in step 2 can be done by running this command in Raspberry Pi:

See: http://www.commandlinefu.com/commands/view/5427/get-your-external-ip-address

NumPy’s ndarray indexing

In NumPy a new kind of array is provided: n-dimensional array or ndarray. It’s usually fixed-sized and accepts items of the same type and size. For example, to define a 2×3 matrix:

When indexing ndarray, it supports “array indexing” other than single element indexing.  (See http://docs.scipy.org/doc/numpy/user/basics.indexing.html)

It is possible to index arrays with other arrays for the purposes of selecting lists of values out of arrays into new arrays. There are two different ways of accomplishing this. One uses one or more arrays of index values. The other involves giving a boolean array of the proper shape to indicate the values to be selected. Index arrays are a very powerful tool that allow one to avoid looping over individual elements in arrays and thus greatly improve performance.

So you basically can do the following:

Besides, when you do equals operation on ndarrays, another ndarray is returned by comparing each element:

Statistics of insurance sold on Taobao.com on Valentine’s Day

On Feb. 14th Taobao launched a campaign to sell insurance products, which promises 7% yearly interest rate. The sales data is public, so I wrote a script to crawl them down and did a brief study on this data. Here’re the results.

On that day (actually sold out in less than two hours in total) more than 40,000 people participated, resulting a total sales of almost one billion CNY (the exact number: 980,270,000 CNY). Two companies participated in this sales campaign: Zhujiang and Tian’an. The sales statistics are:

Zhujiang Tian’an Total
# of Customers 13831 29092 42923
Sales mean (k CNY) 24.922059 21.847003 22.837872
Sales min (k CNY) 1 1 1
Sales 25% (k CNY) 1 2 2
Sales 50% (k CNY) 10 10 10
Sales 75% (k CNY) 20 25 22
Sales max (k CNY) 1000 900 1000
Sales total (k CNY) 344697 635573 980270

The histograms of how many people pay for each amount.

le100kgt100kle100k

Zhujiang was extremely popular: in 2 minutes and 56 seconds it reached a sales of 200,212,000 CNY, that’s more than 1 million CNY sales PER SECOND! Indeed Chinese are crazy about online shopping. 😀

MapReduce in MongoDB

http://docs.mongodb.org/manual/core/map-reduce/

http://docs.mongodb.org/manual/reference/command/mapReduce/

The MapReduce code I used to analyze the 20 million hotel reservation records:

 

Debugging Apache configuration files on Mac OS X

Sometimes I make some modification to my site’s apache config and apache stops functioning, but I cannot get enough information from /var/log/apache2/error.log. Here’s a command that will tell you where the problem is:

It starts apache for debugging and prints out the messages on screen, so that hopefully you can find something useful.