Category: Tech Blog

Here you can find my random notes on software design and development, software engineering, coding, and all other tech-related stuff.

File editing within a docker container with cat

Published 2023/01/12 by Daoyuan Li

$ cat > filename
Copy or write some lines here
New lines are accepted
Ctrl-D to exit and savefile
$ cat filename
Copy or write some lines here
New lines are accepted

Leave a Comment

Effective Data Science Team Leadership: Challenges

Published 2023/01/08 by Daoyuan Li

Team leaders play a vital role in the success of any team, whether it’s a small start-up or a large corporation. A team leader is like a conductor leading an orchestra. Just as a conductor brings together a group of musicians with different instruments and backgrounds, a team leader brings together a group of individuals with different skills and expertise. The conductor’s job is to create a cohesive and harmonious piece of music, and the team leader’s job is to create a cohesive and harmonious team.

Like a conductor, a team leader must have a clear vision of what they want to achieve and be able to communicate that vision to their team. They must also be able to listen to the needs and concerns of their team members and adapt their approach as needed. Just as a conductor may need to make adjustments to the tempo or dynamics of a piece of music based on the capabilities of the musicians, a team leader may need to make adjustments to their management style based on the needs and skills of their team.

A team leader must also be able to motivate and inspire their team, just as a conductor must be able to inspire the musicians to give their best performance. This may involve setting clear goals and expectations, providing constructive feedback, and recognizing and rewarding good performance.

Finally, a team leader must be able to bring out the best in their team, just as a conductor must be able to bring out the best in their musicians. This may involve identifying and addressing any issues or conflicts within the team, as well as fostering a positive and inclusive work environment.

The same principles of team leadership that apply to an orchestra also apply to businesses. Good business leaders have a clear vision for their company and are able to communicate that vision to their team. They are able to motivate and inspire their employees, and bring out the best in them. They are also able to adapt to changing circumstances and make adjustments as needed.

The impact of good business leaders cannot be overstated. Good business leaders have the ability to take a company to new heights, driving innovation and growth. They are able to create a positive and productive work environment, which can lead to increased employee satisfaction and retention. Good business leaders are also able to make sound decisions that drive the success of the company. One example of a good business leader is Jeff Bezos, the Founder of Amazon. Bezos has been able to take Amazon from a small online bookstore to a global e-commerce giant. He has a clear vision for the company and has consistently been able to adapt to changing circumstances and stay ahead of the competition. Bezos has also fostered a culture of innovation at Amazon, and has played a key role in driving the company’s growth.

On the other hand, bad business leaders can have a detrimental impact on a company. They may lack a clear vision or be unable to effectively communicate that vision to their team. They may also be inflexible and resistant to change, which can stifle innovation and growth. Bad business leaders may also create a negative work environment, leading to low employee morale and high turnover. Ultimately, bad business leaders can make poor decisions that can lead to the failure of the company. An example of a bad business leader is Elizabeth Holmes, the former CEO of Theranos. Holmes had a vision for revolutionizing the healthcare industry with her company’s technology, but was unable to effectively communicate that vision to her team and to gear the team towards a realistic vision. She was also resistant to change and unwilling to listen to the concerns of her team, leading to a number of problems within the company. Ultimately, Holmes’ poor leadership contributed to the failure of Theranos.

In conclusion, good business leaders are essential for the success of a company. They are able to drive innovation and growth, create a positive work environment, and make sound decisions. On the other hand, bad business leaders can have a detrimental impact on a company, leading to failure. The impact of good and bad business leaders can be seen in real-life examples throughout history and it is clear that strong leadership is crucial for the success of any business.

Leading software engineering teams

Not every leader has the opportunity to lead large businesses, but software engineering leaders are less difficult to find. However, leading a small or medium-sized software engineering team presents its own set of unique challenges and opportunities. It is also very different from leading other types of teams. Here, we explore some of the key characteristics of software engineering teams and how they differ from other types of teams, and we will discuss some of the key challenges and considerations for leaders of these teams.

One of the most significant differences between software engineering teams and other types of teams is the skill level and focus of team members. Software engineering team members are typically highly skilled and deeply knowledgeable about their work, with a focus on problem-solving and creativity. As a leader, it is important to recognize and respect the skills and expertise of your team members, and to create an environment that allows them to focus on these areas. This may involve setting clear goals and objectives, providing resources and support, and encouraging collaboration and innovation within the team.

Another key difference between software engineering teams and other types of teams is the organizational structure. Many software engineering teams operate with a flat organizational structure, with little or no hierarchy. This allows team members to collaborate and communicate more easily, and can be a key factor in the team’s success. As a leader, it is important to embrace this flat structure and to foster an open and collaborative team culture. This may involve facilitating communication, resolving conflicts, and building trust and collaboration within the team.

Software development projects are also typically very complex, with many different components and dependencies. As a leader, it is important to have a deep understanding of the project and to be able to effectively communicate this understanding to your team. You should also be able to anticipate and plan for potential challenges and roadblocks, and be able to adapt and pivot as needed.

Another major difference between software engineering teams and other types of teams is the rapid pace of technological change in the field. New technologies and approaches are constantly emerging, and it is important for leaders of software engineering teams to stay up to date with these developments and to be able to effectively incorporate them into their team’s work. This may require continuous learning and professional development efforts, such as attending conferences, taking online courses, and reading industry publications.

In addition to these technical challenges, leaders of software engineering teams also need to be able to effectively manage stakeholder expectations and team dynamics. Software development projects often have many stakeholders, with different expectations and priorities, and it is important for leaders to be able to effectively manage these expectations and to communicate effectively with stakeholders to ensure that the project stays on track and meets its goals. Managing team dynamics is also critical, as software engineering teams can be diverse, with team members from different backgrounds and with different personalities. As a leader, it is important to be able to effectively manage team dynamics and to foster a positive and inclusive team culture.

By understanding the characteristics of software engineering teams and the key challenges and considerations for leaders of these teams, it is obvious that leading a software engineering team is a unique and challenging responsibility, requiring a range of technical, communication, and management skills.

Leading data science teams

Leading a data science team is once again different from leading a generic software engineering team in several key ways. There are several challenges that leaders may face when transitioning from leading a software engineering team to a data science team. Some of the key challenges include:

Different decision-making processes: Data science teams often rely on data-driven decision making, which can be different from the decision-making processes used in software engineering teams. One key aspect of data-driven decision making in a data science team is the ability to efficiently explore and analyze large amounts of data. In many cases, the data sets that data science teams work with can be quite large, and it can be challenging to effectively analyze and extract insights from them. Leaders may need to learn how to effectively use data to inform their decisions and communicate the results of their analyses to their teams and stakeholders.
Different technical skills: Data science teams typically require a different set of technical skills than software engineering teams. While software engineering teams focus on building and maintaining software applications, data science teams focus on collecting, analyzing, interpreting, and modelling data. To efficiently explore and analyze large data sets, data science teams often use a variety of tools and techniques. One common approach is to use big data technologies, such as Hadoop or Spark, which are designed to handle large data sets and enable distributed processing. These technologies can help data science teams to quickly and efficiently process and analyze data, even when it is too large to fit on a single machine. In addition to big data technologies, data science teams may also use specialized data exploration and visualization tools. These tools can help team members quickly and easily visualize and understand data, and identify patterns and trends that may not be immediately apparent. As a result, leaders may need to gain new technical skills, such as data analysis and machine learning, in order to effectively lead their teams.
Different team structures: Data science teams may have different structures and workflows than software engineering teams. For example, data science teams may be more decentralized, with team members working on a variety of different projects at the same time. This can be a challenge for leaders who are used to more traditional, hierarchical team structures.
Different cultural norms: Data science teams may also have different cultural norms and expectations than software engineering teams. For example, data science teams may place a greater emphasis on collaboration and cross-functional teamwork, while software engineering teams may be more focused on individual contributions. Leaders may need to adapt to these different cultural norms in order to effectively lead their teams. Data science teams often work closely with other teams and departments within an organization, such as product management, marketing, and sales. As the leader of a data science team, you will need to be able to effectively collaborate with these other teams, and be able to bridge the gap between technical and non-technical domains.
Different stakeholder expectations: Data science teams may have different stakeholders with different expectations than software engineering teams. For example, data science teams may be more closely aligned with business functions, and may need to work more closely with business leaders to understand their needs and goals. Leaders may need to adapt to these different stakeholder expectations. As the leader of a data science team, you will need to be able to analyze large sets of data and use statistical and machine learning techniques to extract insights and inform business decisions. This requires a strong understanding of data analysis and visualization tools, as well as an ability to communicate complex technical concepts to non-technical stakeholders.
Different team dynamics: Data science teams may have different team dynamics and communication styles than software engineering teams. For example, data science teams may involve more collaboration and cross-functional teamwork, which can require different leadership approaches and communication styles. Leaders may need to adapt to these different team dynamics in order to effectively lead their teams.
Different tools and technologies: Data science teams may use different tools and technologies than software engineering teams. For example, data science teams may rely more heavily on data analysis and visualization tools, while software engineering teams may use more traditional software development tools. Leaders may need to learn how to effectively use these different tools and technologies in order to lead their teams effectively.
Managing data privacy and security: Data science teams often work with large volumes of sensitive data, and may need to ensure that this data is handled securely and in compliance with relevant regulations. Leaders may need to learn about data privacy and security best practices in order to effectively manage these risks.
Managing team members with diverse backgrounds: Data science teams may have team members with diverse backgrounds and expertise, including computer science, statistics, mathematics, and domain-specific knowledge. Leaders may need to learn how to effectively manage and motivate team members with these diverse backgrounds.
Staying up to date with the latest engineering in the field: The field of data science is rapidly evolving, with new tools and technologies emerging constantly. Leaders may need to make a concerted effort to stay up to date with these developments in order to effectively lead their teams. This may require continuous learning and professional development efforts, such as attending conferences, taking online courses, and reading academic as well as industry publications. While it is difficult to quantify exactly how fast the field of data science is progressing, as it is a broad and rapidly-evolving field that encompasses a wide range of technologies and techniques. However, it is clear that the field is experiencing rapid growth and change. One way to measure the pace of change in the field is to look at the growth in the number of job openings for data scientists. According to data from the U.S. Bureau of Labor Statistics, employment of data scientists is projected to grow by 11% between 2019 and 2029, which is much faster than the average for all occupations. This rapid growth is a reflection of the increasing demand for data-driven insights and decision-making in a wide range of industries. Another way to measure the pace of change in the field is to look at the rate of development of new technologies and techniques. For example, the field of machine learning, which is a key component of data science, has seen rapid progress in recent years. In the past decade, there have been significant advances in machine learning algorithms, such as deep learning and reinforcement learning, which have enabled new applications and capabilities. Overall, it is clear that the field of data science is experiencing rapid growth and change, and it is important for data science teams to stay up-to-date and continuously learn in order to remain competitive.

To sum up, leading a data science team comes with its own set of challenges, including the need to stay up to date with rapid advancements in the field and to have a strong understanding of technical aspects of data science. Data science projects can also be complex and interdisciplinary, requiring effective organization and management. In my next blog post, I will discuss strategies for building and leading a successful data science team, including how to overcome these unique challenges.

Leave a Comment

GPTChat-aided Blogging

Published 2023/01/07 by Daoyuan Li

This post details how the earlier blog post was written.

My prompt

Response from ChatGPT

Here it goes!

Hi, can you help me writing blog post?

Sure, I can help you write a blog post. What is the topic of your blog post?

I was going through my previous posts, and I didn’t even remember I wrote them, and I forgot the books I’ve read. Revisiting my blog is like travelling back in time and rediscovering myself.

Here is a draft of your blog post on revisiting your old blog posts:

Revisiting My Old Blog Posts: A Journey Through Time

As I was going through my old blog posts, I was surprised to see how much I had forgotten about the books I had read and the things I had written about. It was like travelling back in time and rediscovering a part of myself that I had forgotten.

That’s a good start, can you add more depth to it?

Sure, here is a revised version of your blog post with more depth:

Revisiting My Old Blog Posts: A Journey Through Time

But revisiting my old blog posts was not just about reminiscing about the past. It was also an opportunity for me to see how far I had come as a reader and a writer. I could see the progress I had made and the things I had learned since I first started blogging. I was able to identify my strengths and weaknesses as a writer, and I gained a deeper understanding of my own writing style and voice.

Could you complete it with some examples?

Sure, here is a revised version of your blog post with some examples:

Revisiting My Old Blog Posts: A Journey Through Time

For example, I came across a post where I had written about "The Alchemist" by Paulo Coelho. As I reread my thoughts on the book, I was struck by how much my interpretation of the story had changed since I first read it. When I first read the book, I was impressed by the central message of following your dreams. However, upon revisiting the book and my old blog post, I realized that the book had also touched upon deeper themes such as the power of personal growth and the importance of listening to one's own heart.

In addition to personal growth, revisiting my old blog posts also reminded me of the value of consistency and dedication. I realized that I had been consistently reading and writing for a number of years, and that this dedication had played a significant role in my development as a reader and a writer. It reminded me that the journey of self-improvement is a long-term process, and that small steps taken consistently can lead to significant progress over time.

Replace the “The Alchemist” example with the book “First Love, Last Rites”.

Sure, here is a revised version of your blog post with the book "First Love, Last Rites" as the example:

Revisiting My Old Blog Posts: A Journey Through Time

For example, I came across a post where I had written about "First Love, Last Rites" by Ian McEwan. As I reread my thoughts on the book, I was struck by how much my interpretation of the story had changed since I first read it. When I first read the book, I was impressed by the vivid and poetic language used by the author. However, upon revisiting the book and my old blog post, I realized that the book had also touched upon deeper themes such as the power of first love and the fragility of life.

Add a few more example on writing skills.

Sure, here is a revised version of your blog post with additional examples of your writing skills:

Revisiting My Old Blog Posts: A Journey Through Time

For example, I noticed that my writing had become more structured and focused over the years. In my earlier posts, I had a tendency to ramble and include tangential thoughts that distracted from the main message. However, as I reread my more recent posts, I could see that I had learned to be more concise and stick to the main points. I had also improved my ability to present complex ideas in a clear and concise manner, as demonstrated in a post where I analyzed the themes of identity and belonging in "The Great Gatsby".

Replace “The Great Gatsby” example with “The unconsciousness rules our lives”

Sure, here is a revised version of your blog post with "The unconsciousness rules our lives" as the example:

Revisiting My Old Blog Posts: A Journey Through Time

No, stick with the previous version with “The great gatsby”

Sure, here is the previous version of your blog post with "The Great Gatsby" as the example:

Revisiting My Old Blog Posts: A Journey Through Time

That’s not the correct version. Let’s continue with this one: Revisiting My Old Blog Posts: A Journey Through Time As I was going through my old blog posts, I was surprised to see how much I had forgotten about the books I had read and the things I had written about. It was like travelling back in time and rediscovering a part of myself that I had forgotten. For example, I came across a post where I had written about “First Love, Last Rites” by Ian McEwan. As I reread my thoughts on the book, I was struck by how much my interpretation of the story had changed since I first read it. When I first read the book, I was impressed by the vivid and poetic language used by the author. However, upon revisiting the book and my old blog post, I realized that the book had also touched upon deeper themes such as the power of first love and the fragility of life. Reading my old blog posts brought back memories of the time when I first read those books and shared my thoughts on them. It was interesting to see how my opinions and perspectives on certain topics had changed over the years. I realized that I had grown as a person, and my views on various subjects had evolved as I gained more knowledge and life experience. But revisiting my old blog posts was not just about reminiscing about the past. It was also an opportunity for me to see how far I had come as a reader and a writer. I could see the progress I had made and the things I had learned since I first started blogging. I was able to identify my strengths and weaknesses as a writer, and I gained a deeper understanding of my own writing style and voice. For example, I noticed that my writing had become more structured and focused over the years. In my earlier posts, I had a tendency to ramble and include tangential thoughts that distracted from the main message. However, as I reread my more recent posts, I could see that I had learned to be more concise and stick to the main points. In addition to personal growth, revisiting my old blog posts also reminded me of the value of consistency and dedication. I realized that I had been consistently reading and writing for a number of years, and that this dedication had played a significant role in my development as a reader and a writer. It reminded me that the journey of self-improvement is a long-term process, and that small steps taken consistently can lead to significant progress over time. Overall, revisiting my old blog posts was a meaningful and enriching experience. It reminded me of the importance of reflection and the value of looking back at where we have been in order to move forward. It also encouraged me to continue learning, growing, and writing in the future.

Sure, here are some additional examples of your writing skills:

Revisiting My Old Blog Posts: A Journey Through Time

Overall, revisiting my old blog posts was a meaningful and enriching

Attribute part of my growth or improvement to life experiences such as meeting different persons, traveling, and life changing events such as losing my mother to cancer.

Sure, here is a revised version of your blog post that attributes your growth and improvement to life experiences:

Revisiting My Old Blog Posts: A Journey Through Time

Reading my old blog posts brought back memories of the time when I first read those books and shared my thoughts on them. It was interesting to see how my opinions and perspectives on certain topics had changed over the years. I realized that I had grown as a person, and my views on various subjects had evolved as I gained more knowledge and life experience. Meeting different people, traveling to new places, and experiencing life-changing events such as losing my mother to cancer had all contributed to my growth and development as an individual.

In addition to personal growth, revisiting my old blog posts also reminded me of the value of consistency and dedication. I realized that I had

please continue

Certainly, here is the remainder of your revised blog post:

been consistently reading and writing for a number of years, and that this dedication had played a significant role in my development as a reader and a writer. It reminded me that the journey of self-improvement is a long-term process, and that small steps taken consistently can lead to significant progress over time.

Overall, revisiting my old blog posts was a meaningful and enriching experience. It reminded me of the importance of reflection and the value of looking back at where we have been in order to move forward. It also encouraged me to continue learning, growing, and writing in the future. By acknowledging my past experiences and my progress as a reader and a writer, I am motivated to continue on this journey of self-discovery and self-improvement.

Now add a few paragraphs to attribute part of my growth or improvement to experiences such as meeting different persons, traveling, and life changing events such as losing my mother to cancer.

Certainly, here is a revised version of your blog post that attributes your growth and improvement to life experiences:

Revisiting My Old Blog Posts: A Journey Through Time

But my growth and development as an individual was not solely the result of reading and writing. Meeting different people, traveling to new places, and experiencing life-changing events such as losing my mother to cancer had all played a significant role in shaping my perspectives and helping me to grow as a person. These experiences helped me to see the world from different angles and to gain a deeper understanding of myself and others.

Rev

continue

Certainly, here is the remainder of your revised blog post:

isiting my old blog posts was not just about reminiscing about the past. It was also an opportunity for me to see how far I had come as a reader and a writer. I could see the progress I had made and the things I had learned since I first started blogging. I was able to identify my strengths and weaknesses as a writer, and I gained a deeper understanding of my own writing style and voice.

Add a new paragraph at the end with some action points, such as writing blog posts more frequently, and reading more books.

Certainly, here is a revised version of your blog post with an additional paragraph at the end with action points:

Revisiting My Old Blog Posts: A Journey Through Time

Revisiting my old blog posts was not just about reminiscing about the past. It was also an opportunity for me to see how far I had come as a reader and a writer. I could see the progress I had made and the things I had learned since I first started blogging. I was able to identify my strengths and weaknesses as a writer, and I gained a deeper understanding of my own writing style and voice.

continue

Certainly, here is the remainder of your revised blog post:

the journey of self-improvement is a long-term process, and that small steps taken consistently can lead to significant progress over time.

In light of this experience, I have set some action points for myself to continue growing and improving as a reader and a writer. These include writing blog posts more frequently, reading more books from a diverse range of authors and genres, and seeking out new experiences that will challenge and enrich me. I am excited to see where this journey of self-improvement will take me next.

Good enough, can you further polish this text to sound a little bit more sophisticated?

Certainly, here is a revised version of your blog post that aims to sound more sophisticated:

Reflecting on Past Blog Posts: A Retrospective Journey

As I perused my older blog posts, I was struck by how much I had forgotten about the books I had read and the thoughts I had previously expressed. It was almost as if I were traveling back in time and rediscovering a part of myself that had been relegated to the recesses of my memory.

One post that particularly stood out to me was my review of Ian McEwan's "First Love, Last Rites". Upon rereading my thoughts on the book, I was struck by the extent to which my interpretation of the story had evolved since my initial reading. Initially, I had been enamored with the vivid and poetic language employed by the author. However, upon revisiting the text and my past musings, I came to realize that the novel also delved into deeper themes such as the transformative power of first love and the impermanence of life.

Rereading my old blog posts not only allowed me to reminisce about the past, but also provided me with the opportunity to assess my personal growth as both a reader and a writer. It was interesting to see how my opinions and perspectives on various subjects had changed over the years, a reflection of the ways in which I had grown and developed as an individual. This growth was not solely the result of my reading and writing endeavors, but was also influenced by the people I have met, the places I have visited, and the life-changing events, such as the loss of my mother to cancer, that have shaped my worldview.

In addition to personal growth, revisiting my old blog posts also emphasized the importance of consistency and dedication. I was reminded of the fact that I had been consistently reading and writing for a number of years, and that this dedication had played a significant role in my development as a reader and a writer. It was a reminder that the journey of self-improvement is a long-term process, and that small, consistent steps can lead to significant progress over time.

In conclusion, reflecting on my old blog posts was a meaningful and enriching experience. It emphasized the value of reflection and the importance of looking back in order to move forward. It also encouraged me to continue learning, growing, and writing in the future. In light of this retrospective journey, I have set some action points for myself, including writing blog posts more frequently, reading a diverse range of books, and seeking out new experiences that will challenge and enrich me. I am eager to see where this journey of self-improvement will take me next.

Great, am I allowed to publish this post on my website?

You are welcome to use the revised version of your blog post on your website. I am an AI language model and do not hold any copyrights to the content that I generate, so you are free to use it as you see fit. However, it is important to remember that the content you publish on your website should always be accurate and respectful. If you are using information from other sources in your blog post, it is essential to properly cite those sources to give credit to the original authors and to ensure the accuracy of your information. Additionally, you should ensure that your content does not infringe on the copyright of others. I hope that this helps!

Can you make it longer with more examples?

Certainly, here is a revised version of your blog post that includes additional examples:

Reflecting on Past Blog Posts: A Retrospective Journey

Another post that I revisited was my analysis of F. Scott Fitzgerald's "The Great Gatsby". As I reread my thoughts on the novel, I was impressed by my own ability to present complex ideas in a clear and concise manner. I had examined the themes of identity and belonging in the text, and had drawn connections to the broader societal context in which the novel was set. This level of analysis and synthesis was a testament to the progress I had made as a reader and a writer.

continue

Certainly, here is the remainder of your revised blog post:

For example, I am planning to take on the challenge of reading at least one book per month from a different genre or by a diverse range of authors. This will not only expose me to new ideas and perspectives, but will also help me to continue improving my reading comprehension and critical thinking skills. I am also committed to writing more frequently, as this will help me to hone my writing skills and to better express my thoughts and ideas. And finally, I am excited to seek out new experiences that will challenge and enrich me, such as traveling to new places or volunteering in my community. I believe that these actions will not only help me to continue growing and improving as a person, but will also make for more interesting and thought-provoking content for my blog.

Rewrite “The Great Gatsby” paragraph with something from “Blink: the power of thinking without thinking”

Certainly, here is a revised version of your blog post with the "The Great Gatsby" paragraph replaced with content from "Blink: The Power of Thinking Without Thinking":

Reflecting on Past Blog Posts: A Retrospective Journey

Another post that I revisited was my analysis of Malcolm Gladwell's "Blink: The Power of Thinking Without Thinking". In this post, I had examined the concept of "thin slicing" and its implications for decision making. I had explored the ways in which our unconscious mind is able to process vast amounts of information in a split second, and how this can sometimes lead to more accurate decisions than if we were to consciously analyze all of the available data. This post was a testament to my growth as a reader and a writer, as I had been able to clearly articulate and analyze complex ideas.

continue

Certainly, here is the remainder of your revised blog post:

Leave a Comment

Migrating self-hosted WordPress

Published 2023/01/07 by Daoyuan Li

When I started this website in 2009, I was using Joomla. Then at some point I switch to WordPress because it was much easier to manage my posts. And it was hosted at one.com, where I initially registered this domain name. However, the network and service wasn’t really stable and I had a lot of hiccups back then. As a result, in 2017 I migrated the website to Scaleway.com, which offers very cheap virtual machines and small bare metal instances.

Another reason why I switched was because I was also running a real-time crypto price crawling and algorithmic trading service with a few servers, and since my website didn’t have a ton of visitors, I could put it together with other services. Back then a small instance cost only 5 euros per month, and I found it to be more cost-effective than other cloud computing providers like AWS, GCP or Vultr. As time went by, I had more resource intensive services running, such as staking pools which require large volumes. I ended up upgrading those tiny instances, and adding more resources from Scaleway. Then later when I phased out such services and reduced the number of instances, I couldn’t switch back to the smaller instances because either they didn’t exist any more or I couldn’t downgrade the volume size. So in the end I was paying a large monthly bill to run a small WordPress application.

And today I finally managed to spare a few hours reviewing the services I need, and decided it to migrate it back to one.com, which I was already paying anyway. As a side note, the price of one.com subscription also increase by 4x or 5x in 10 years.

Migrating WordPress alone is easy: backup all the files and database, then copy files to one.com, import the database, and setup the DNS on Cloudflare. I asked ChatGPT to draw me this flowchart:

+-------------+                                +-------------+
| Old Server  |   Backup files and database   | Local Server |
+-------------+ ----------------------------> +-------------+
                                                    |
                                                    | Import database
                                                    v
                                               +------------+
                                               | New Server  |
                                               +------------+
                                                    |
                                                    | Update URLs in database
                                                    v
                                               +------------+
                                               | New Server  |
                                               +------------+

During the process, I also tried setting up a local LAMP+WP service with docker-compose (this repo is very useful: https://github.com/nezhar/wordpress-docker-compose) and tried exporting my posts into GoHugo format. In the end I believed it was too much hassle and not worth it.

A few issues I encountered during the migration process:

.htaccess files should be set up properly, otherwise WP complains about 404 for sub pages.
Some WP plugins were outdated (code highlighting, for example), so it caused some page rendering issues. When I replaced the plugin such issues are gone.
File permissions has to be 744 or 755 for the wp-content/uploads folder.
I used nslookup daoyuan.li ns01.one.com to get the IP of one.com’s server, and manually updated it on Cloudflare, since I prefer to use Cloudflare to manage my DNS. In the future I may need to automate this process, in case one.com moves my VM to another server. Or at lease I should monitor the output from ns01.one.com versus kara.ns.cloudflare.com.

Besides migrating WP, I also had a few other websites to migrate. But since they are all static websites, I used Cloudflare Pages to host them and assigned the domains to the corresponding Pages project.

I probably spent 3 to 4 hours migrating everything, and double checking everything works fine. Then I went ahead and terminated all instances, elastic IPs and volumes on Scaleway. That would save me a couple of hundred euros a year. Not bad ROI!

Leave a Comment

Notes to myself: RTX 2070, Cuda, cudnn, caffe, and faceswap

Published 2019/05/19 by Daoyuan Li

Install NVIDIA driver for RTX 2070: https://www.geforce.com/drivers

sudo bash ~/Downloads/NVIDIA-Linux-x86_64-415.27.run

Install CUDA 10.0: https://developer.nvidia.com/cuda-downloads

sudo bash ~/Downloads/cuda_10.0.130_410.48_linux.run

DO NOT re-install the drivers suggested by the CUDA installer:
Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 410.48?
(y)es/(n)o/(q)uit: n

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 415.27       Driver Version: 415.27       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:1F:00.0 Off |                  N/A |
| 36%   31C    P0    N/A /  N/A |      0MiB /  7949MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

CuDNN:

cuDNN Runtime Library for Ubuntu18.04 (Deb)

cuDNN Developer Library for Ubuntu18.04 (Deb)

Caffe:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_cublas_device_LIBRARY (ADVANCED)
linked by target “caffe” in directory /home/dli/Projects/caffe/src/caffe

Upgrade cmake:

$ ~/cmake-3.13.3-Linux-x86_64/bin/cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr ..
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   system
--   thread
--   filesystem
--   chrono
--   date_time
--   atomic
-- Found gflags  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
-- Found glog    (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
-- Found PROTOBUF Compiler: /usr/bin/protoc
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- HDF5: Using hdf5 compiler wrapper to determine CXX configuration
-- Found lmdb    (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
-- Found Snappy  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so)
-- CUDA detected: 10.0
-- Found cuDNN: ver. 7.4.2 found (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- Added CUDA NVCC flags for: sm_75
-- OpenCV found (/usr/share/OpenCV)
-- Found Atlas (include: /usr/include/x86_64-linux-gnu library: /usr/lib/x86_64-linux-gnu/libatlas.so lapack: /usr/lib/x86_64-linux-gnu/liblapack.so
-- NumPy ver. 1.16.0 found (include: /usr/local/lib/python2.7/dist-packages/numpy/core/include)
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   python
-- Detected Doxygen OUTPUT_DIRECTORY: ./doxygen/
--
-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   1.0.0
--   Git               :   1.0-132-g99bd9979
--   System            :   Linux
--   C++ compiler      :   /usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
--
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   ON
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--   USE_OPENCV        :   ON
--   USE_LEVELDB       :   ON
--   USE_LMDB          :   ON
--   USE_NCCL          :   OFF
--   ALLOW_LMDB_NOLOCK :   OFF
--   USE_HDF5          :   ON
--
-- Dependencies:
--   BLAS              :   Yes (Atlas)
--   Boost             :   Yes (ver. 1.65)
--   glog              :   Yes
--   gflags            :   Yes
--   protobuf          :   Yes (ver. 3.0.0)
--   lmdb              :   Yes (ver. 0.9.21)
--   LevelDB           :   Yes (ver. 1.20)
--   Snappy            :   Yes (ver. ..)
--   OpenCV            :   Yes (ver. 3.2.0)
--   CUDA              :   Yes (ver. 10.0)
--
-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_75
--   cuDNN             :   Yes (ver. 7.4.2)
--
-- Python:
--   Interpreter       :   /usr/bin/python2.7 (ver. 2.7.15)
--   Libraries         :   /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.15rc1)
--   NumPy             :   /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.16.0)
--
-- Documentaion:
--   Doxygen           :   /usr/bin/doxygen (1.8.13)
--   config_file       :   /home/dli/Projects/caffe/.Doxyfile
--
-- Install:
--   Install path      :   /usr
--
-- Configuring done
-- Generating done
-- Build files have been written to: /home/dli/Projects/caffe/build
$ make -j16 all
$ make pycaffe
$ sudo make install
[  2%] Built target caffeproto
[ 87%] Built target caffe
[ 89%] Built target upgrade_solver_proto_text
[ 90%] Built target compute_image_mean
[ 90%] Built target caffe.bin
[ 91%] Built target upgrade_net_proto_binary
[ 93%] Built target convert_imageset
[ 94%] Built target extract_features
[ 95%] Built target upgrade_net_proto_text
[ 95%] Built target classification
[ 95%] Built target convert_mnist_data
[ 97%] Built target convert_cifar_data
[ 98%] Built target convert_mnist_siamese_data
[100%] Built target pycaffe
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/share/Caffe/CaffeConfig.cmake
-- Installing: /usr/share/Caffe/CaffeTargets.cmake
-- Installing: /usr/share/Caffe/CaffeTargets-release.cmake
-- Installing: /usr/include/caffe
-- Installing: /usr/include/caffe/test
-- Installing: /usr/include/caffe/test/test_gradient_check_util.hpp
-- Installing: /usr/include/caffe/test/test_caffe_main.hpp
-- Installing: /usr/include/caffe/layers
-- Installing: /usr/include/caffe/layers/cudnn_tanh_layer.hpp
-- Installing: /usr/include/caffe/layers/absval_layer.hpp
-- Installing: /usr/include/caffe/layers/multinomial_logistic_loss_layer.hpp
-- Installing: /usr/include/caffe/layers/dummy_data_layer.hpp
-- Installing: /usr/include/caffe/layers/recurrent_layer.hpp
-- Installing: /usr/include/caffe/layers/scale_layer.hpp
-- Installing: /usr/include/caffe/layers/hdf5_data_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_sigmoid_layer.hpp
-- Installing: /usr/include/caffe/layers/clip_layer.hpp
-- Installing: /usr/include/caffe/layers/hinge_loss_layer.hpp
-- Installing: /usr/include/caffe/layers/mvn_layer.hpp
-- Installing: /usr/include/caffe/layers/relu_layer.hpp
-- Installing: /usr/include/caffe/layers/hdf5_output_layer.hpp
-- Installing: /usr/include/caffe/layers/contrastive_loss_layer.hpp
-- Installing: /usr/include/caffe/layers/lrn_layer.hpp
-- Installing: /usr/include/caffe/layers/accuracy_layer.hpp
-- Installing: /usr/include/caffe/layers/conv_layer.hpp
-- Installing: /usr/include/caffe/layers/infogain_loss_layer.hpp
-- Installing: /usr/include/caffe/layers/im2col_layer.hpp
-- Installing: /usr/include/caffe/layers/base_conv_layer.hpp
-- Installing: /usr/include/caffe/layers/euclidean_loss_layer.hpp
-- Installing: /usr/include/caffe/layers/crop_layer.hpp
-- Installing: /usr/include/caffe/layers/window_data_layer.hpp
-- Installing: /usr/include/caffe/layers/bnll_layer.hpp
-- Installing: /usr/include/caffe/layers/eltwise_layer.hpp
-- Installing: /usr/include/caffe/layers/prelu_layer.hpp
-- Installing: /usr/include/caffe/layers/filter_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_lcn_layer.hpp
-- Installing: /usr/include/caffe/layers/reduction_layer.hpp
-- Installing: /usr/include/caffe/layers/sigmoid_cross_entropy_loss_layer.hpp
-- Installing: /usr/include/caffe/layers/swish_layer.hpp
-- Installing: /usr/include/caffe/layers/slice_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_softmax_layer.hpp
-- Installing: /usr/include/caffe/layers/reshape_layer.hpp
-- Installing: /usr/include/caffe/layers/silence_layer.hpp
-- Installing: /usr/include/caffe/layers/sigmoid_layer.hpp
-- Installing: /usr/include/caffe/layers/power_layer.hpp
-- Installing: /usr/include/caffe/layers/spp_layer.hpp
-- Installing: /usr/include/caffe/layers/exp_layer.hpp
-- Installing: /usr/include/caffe/layers/pooling_layer.hpp
-- Installing: /usr/include/caffe/layers/input_layer.hpp
-- Installing: /usr/include/caffe/layers/data_layer.hpp
-- Installing: /usr/include/caffe/layers/lstm_layer.hpp
-- Installing: /usr/include/caffe/layers/neuron_layer.hpp                                                                                                                                     
-- Installing: /usr/include/caffe/layers/split_layer.hpp
-- Installing: /usr/include/caffe/layers/threshold_layer.hpp
-- Installing: /usr/include/caffe/layers/base_data_layer.hpp
-- Installing: /usr/include/caffe/layers/log_layer.hpp
-- Installing: /usr/include/caffe/layers/loss_layer.hpp
-- Installing: /usr/include/caffe/layers/rnn_layer.hpp
-- Installing: /usr/include/caffe/layers/elu_layer.hpp
-- Installing: /usr/include/caffe/layers/memory_data_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_relu_layer.hpp
-- Installing: /usr/include/caffe/layers/tanh_layer.hpp
-- Installing: /usr/include/caffe/layers/flatten_layer.hpp
-- Installing: /usr/include/caffe/layers/dropout_layer.hpp
-- Installing: /usr/include/caffe/layers/bias_layer.hpp
-- Installing: /usr/include/caffe/layers/softmax_loss_layer.hpp
-- Installing: /usr/include/caffe/layers/deconv_layer.hpp
-- Installing: /usr/include/caffe/layers/inner_product_layer.hpp
-- Installing: /usr/include/caffe/layers/batch_norm_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_conv_layer.hpp
-- Installing: /usr/include/caffe/layers/parameter_layer.hpp
-- Installing: /usr/include/caffe/layers/tile_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_deconv_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_pooling_layer.hpp
-- Installing: /usr/include/caffe/layers/python_layer.hpp
-- Installing: /usr/include/caffe/layers/embed_layer.hpp
-- Installing: /usr/include/caffe/layers/image_data_layer.hpp
-- Installing: /usr/include/caffe/layers/batch_reindex_layer.hpp
-- Installing: /usr/include/caffe/layers/softmax_layer.hpp
-- Installing: /usr/include/caffe/layers/concat_layer.hpp
-- Installing: /usr/include/caffe/layers/cudnn_lrn_layer.hpp
-- Installing: /usr/include/caffe/layers/argmax_layer.hpp
-- Installing: /usr/include/caffe/blob.hpp
-- Installing: /usr/include/caffe/util
-- Installing: /usr/include/caffe/util/upgrade_proto.hpp
-- Installing: /usr/include/caffe/util/blocking_queue.hpp
-- Installing: /usr/include/caffe/util/db.hpp
-- Installing: /usr/include/caffe/util/db_lmdb.hpp
-- Installing: /usr/include/caffe/util/insert_splits.hpp
-- Installing: /usr/include/caffe/util/format.hpp
-- Installing: /usr/include/caffe/util/im2col.hpp
-- Installing: /usr/include/caffe/util/gpu_util.cuh
-- Installing: /usr/include/caffe/util/signal_handler.h
-- Installing: /usr/include/caffe/util/cudnn.hpp
-- Installing: /usr/include/caffe/util/hdf5.hpp
-- Installing: /usr/include/caffe/util/io.hpp
-- Installing: /usr/include/caffe/util/nccl.hpp
-- Installing: /usr/include/caffe/util/rng.hpp
-- Installing: /usr/include/caffe/util/db_leveldb.hpp
-- Installing: /usr/include/caffe/util/mkl_alternate.hpp
-- Installing: /usr/include/caffe/util/benchmark.hpp
-- Installing: /usr/include/caffe/util/device_alternate.hpp
-- Installing: /usr/include/caffe/util/math_functions.hpp
-- Installing: /usr/include/caffe/sgd_solvers.hpp
-- Installing: /usr/include/caffe/layer.hpp
-- Installing: /usr/include/caffe/net.hpp
-- Installing: /usr/include/caffe/syncedmem.hpp
-- Installing: /usr/include/caffe/solver.hpp
-- Installing: /usr/include/caffe/filler.hpp
-- Installing: /usr/include/caffe/layer_factory.hpp                                                                                                                                            [0/863]
-- Installing: /usr/include/caffe/data_transformer.hpp
-- Installing: /usr/include/caffe/solver_factory.hpp
-- Installing: /usr/include/caffe/parallel.hpp
-- Installing: /usr/include/caffe/common.hpp
-- Installing: /usr/include/caffe/internal_thread.hpp
-- Installing: /usr/include/caffe/caffe.hpp
-- Installing: /usr/include/caffe/proto/caffe.pb.h
-- Installing: /usr/lib/x86_64-linux-gnu/libcaffe.so.1.0.0
-- Set runtime path of "/usr/lib/x86_64-linux-gnu/libcaffe.so.1.0.0" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/lib/x86_64-linux-gnu/libcaffe.so
-- Installing: /usr/lib/x86_64-linux-gnu/libcaffeproto.a
-- Installing: /usr/python/caffe/proto/caffe_pb2.py
-- Installing: /usr/python/caffe/proto/__init__.py
-- Installing: /usr/bin/caffe
-- Set runtime path of "/usr/bin/caffe" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/compute_image_mean
-- Set runtime path of "/usr/bin/compute_image_mean" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/convert_imageset
-- Set runtime path of "/usr/bin/convert_imageset" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/extract_features
-- Set runtime path of "/usr/bin/extract_features" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/upgrade_net_proto_binary
-- Set runtime path of "/usr/bin/upgrade_net_proto_binary" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/upgrade_net_proto_text
-- Set runtime path of "/usr/bin/upgrade_net_proto_text" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/upgrade_solver_proto_text
-- Set runtime path of "/usr/bin/upgrade_solver_proto_text" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/convert_cifar_data
-- Set runtime path of "/usr/bin/convert_cifar_data" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/classification
-- Set runtime path of "/usr/bin/classification" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/convert_mnist_data
-- Set runtime path of "/usr/bin/convert_mnist_data" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/bin/convert_mnist_siamese_data
-- Set runtime path of "/usr/bin/convert_mnist_siamese_data" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"
-- Installing: /usr/python/classify.py
-- Installing: /usr/python/detect.py
-- Installing: /usr/python/draw_net.py
-- Installing: /usr/python/requirements.txt
-- Installing: /usr/python/train.py
-- Up-to-date: /usr/python/caffe
-- Installing: /usr/python/caffe/io.py
-- Installing: /usr/python/caffe/pycaffe.py
-- Up-to-date: /usr/python/caffe/proto
-- Installing: /usr/python/caffe/proto/caffe_pb2.py
-- Installing: /usr/python/caffe/proto/__init__.py
-- Installing: /usr/python/caffe/coord_map.py
-- Installing: /usr/python/caffe/classifier.py
-- Installing: /usr/python/caffe/__init__.py
-- Installing: /usr/python/caffe/imagenet
-- Installing: /usr/python/caffe/imagenet/ilsvrc_2012_mean.npy
-- Installing: /usr/python/caffe/net_spec.py
-- Installing: /usr/python/caffe/detector.py
-- Installing: /usr/python/caffe/draw.py
-- Installing: /usr/python/caffe/_caffe.so
-- Set runtime path of "/usr/python/caffe/_caffe.so" to "/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/hdf5/serial:/usr/local/cuda-10.0/lib64"

face_swap

$ git diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index cb34534..762363f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -69,21 +69,30 @@ find_package(OpenCV REQUIRED highgui imgproc imgcodecs calib3d photo)
 # dlib
 find_package(dlib REQUIRED)

+if(DEFINED CAFFE_DIR)
+    list(APPEND CMAKE_PREFIX_PATH ${CAFFE_DIR})
+endif()
+
 # Caffe
 find_package(Caffe REQUIRED)

+if(DEFINED HDF5_DIR)
+    list(APPEND CMAKE_MODULE_PATH ${HDF5_DIR})
+endif()
+
 # HDF5
 if(MSVC)
   # Find HDF5 using it's hdf5-config.cmake file with MSVC
   if(DEFINED HDF5_DIR)
     list(APPEND CMAKE_MODULE_PATH ${HDF5_DIR})
   endif()
-  find_package(HDF5 COMPONENTS C HL CXX REQUIRED)
   set(HDF5_LIBRARIES hdf5-shared hdf5_cpp-shared)
   set(HDF5_HL_LIBRARIES hdf5_hl-shared)
 else()
   #find_package(HDF5 COMPONENTS HL REQUIRED)
+  # find_package(HDF5 COMPONENTS C CXX HL REQUIRED)
   find_package(HDF5 COMPONENTS C CXX HL REQUIRED)
+  # find_package(HDF5 COMPONENTS C HL CXX REQUIRED PATHS /usr/lib/x86_64-linux-gnu/hdf5/serial)
 endif()
 #find_package(HDF5 REQUIRED CXX)

@@ -132,23 +141,23 @@ endif()
 set(FACE_SWAP_TARGETS face_swap iris_sfs)
 export(TARGETS ${FACE_SWAP_TARGETS}
   FILE "${PROJECT_BINARY_DIR}/face_swap-targets.cmake")
-
+
 # Export the package for use from the build-tree
 # (this registers the build-tree with a global CMake-registry)
 export(PACKAGE face_swap)
-
+
 # Create config files
 configure_file(cmake/face_swap-config.cmake.in
   "${PROJECT_BINARY_DIR}/face_swap-config.cmake" @ONLY)
 configure_file(cmake/face_swap-config-version.cmake.in
   "${PROJECT_BINARY_DIR}/face_swap-config-version.cmake" @ONLY)
-
+
 # Install config files
 install(FILES
   "${PROJECT_BINARY_DIR}/face_swap-config.cmake"
   "${PROJECT_BINARY_DIR}/face_swap-config-version.cmake"
   DESTINATION "cmake" COMPONENT dev)
-
+
 # Install the export set for use with the install-tree
 install(EXPORT face_swap-targets DESTINATION cmake COMPONENT dev)

diff --git a/face_swap/CMakeLists.txt b/face_swap/CMakeLists.txt
index 1fcb187..ca56ecd 100644
--- a/face_swap/CMakeLists.txt
+++ b/face_swap/CMakeLists.txt
@@ -44,6 +44,7 @@ target_include_directories(face_swap PUBLIC
        ${HDF5_INCLUDE_DIRS}
 )
 target_link_libraries(face_swap PUBLIC
+       uuid
        iris_sfs
        ${OpenCV_LIBS}
        dlib::dlib

$ ~/cmake-3.13.3-Linux-x86_64/bin/cmake -DWITH_BOOST_STATIC=OFF -DBUILD_INTERFACE_PYTHON=ON -DBUILD_SHARED_LIBS=OFF -DBUILD_APPS=ON -DBUILD_TESTS=OFF cmake -DCM
AKE_INSTALL_PREFIX=/usr/local/face_swap -DCMAKE_BUILD_TYPE=Release ..
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   filesystem
--   program_options
--   regex
--   timer
--   thread
--   chrono
--   system
--   date_time
--   atomic
-- Found OpenCV: /usr (found version "3.2.0") found components:  highgui imgproc imgcodecs calib3d photo
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- HDF5: Using hdf5 compiler wrapper to determine CXX configuration
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5_cpp.so;/usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.0.1") found components:  C CXX HL
-- Found Eigen3: /usr/include/eigen3 (Required is at least version "2.91.0")
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread (found version "3.0.0")
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.13") found components:  doxygen missing components:  dot
-- Found SWIG: /usr/bin/swig3.0 (found version "3.0.12")
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found version "3.6.7")
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   python
--   numpy
-- Configuring done
-- Generating done
-- Build files have been written to: /home/dli/Projects/face_swap/build
$ make -j16
$ sudo make install
[ 20%] Built target iris_sfs
[ 66%] Built target face_swap
[ 73%] Built target face_swap_image
[ 80%] Built target face_swap_batch
[ 86%] Built target face_swap_single2many
[ 93%] Built target face_swap_image2video
[100%] Built target face_swap_py
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/face_swap/cmake/face_swap-config.cmake
-- Installing: /usr/local/face_swap/cmake/face_swap-config-version.cmake
-- Installing: /usr/local/face_swap/cmake/face_swap-targets.cmake
-- Installing: /usr/local/face_swap/cmake/face_swap-targets-release.cmake
-- Installing: /usr/local/face_swap/data/images/brad_pitt_01.jpg
-- Installing: /usr/local/face_swap/data/images/bruce_willis_01.jpg
-- Installing: /usr/local/face_swap/lib/libiris_sfs.a
-- Installing: /usr/local/face_swap/include/face_swap/face_swap_export.h
-- Installing: /usr/local/face_swap/lib/libface_swap.a
-- Installing: /usr/local/face_swap/include/face_swap/basel_3dmm.h
-- Installing: /usr/local/face_swap/include/face_swap/cnn_3dmm.h
-- Installing: /usr/local/face_swap/include/face_swap/cnn_3dmm_expr.h
-- Installing: /usr/local/face_swap/include/face_swap/face_seg.h
-- Installing: /usr/local/face_swap/include/face_swap/face_swap_engine.h
-- Installing: /usr/local/face_swap/include/face_swap/face_swap_engine_impl.h
-- Installing: /usr/local/face_swap/include/face_swap/face_swap_c_interface.h
-- Installing: /usr/local/face_swap/include/face_swap/render_utilities.h
-- Installing: /usr/local/face_swap/include/face_swap/utilities.h
-- Installing: /usr/local/face_swap/include/face_swap/face_detection_landmarks.h
-- Installing: /usr/local/face_swap/include/face_swap/landmarks_utilities.h
-- Installing: /usr/local/face_swap/include/face_swap/segmentation_utilities.h
-- Installing: /usr/local/face_swap/bin/face_swap_image
-- Set runtime path of "/usr/local/face_swap/bin/face_swap_image" to ""
-- Installing: /usr/local/face_swap/bin/face_swap_image.cfg
-- Installing: /usr/local/face_swap/bin/face_swap_batch
-- Set runtime path of "/usr/local/face_swap/bin/face_swap_batch" to ""
-- Installing: /usr/local/face_swap/bin/face_swap_batch.cfg
-- Installing: /usr/local/face_swap/bin/face_swap_single2many
-- Set runtime path of "/usr/local/face_swap/bin/face_swap_single2many" to ""
-- Installing: /usr/local/face_swap/bin/face_swap_single2many.cfg
-- Installing: /usr/local/face_swap/bin/face_swap_image2video
-- Set runtime path of "/usr/local/face_swap/bin/face_swap_image2video" to ""
-- Installing: /usr/local/face_swap/bin/face_swap_image2video.cfg
-- Installing: /usr/local/face_swap/interfaces/python/face_swap_py.so
-- Set runtime path of "/usr/local/face_swap/interfaces/python/face_swap_py.so" to ""
$ ls /face_swap/data/
3dmm_cnn_resnet_101.caffemodel          BaselFace.dat                           face_seg_fcn8s_300_no_aug.zip           images/
3dmm_cnn_resnet_101_deploy.prototxt     BaselFaceModel_mod_wForehead_noEars.h5  face_seg_fcn8s.caffemodel               shape_predictor_68_face_landmarks.dat
3dmm_cnn_resnet_101_mean.binaryproto    face_seg_fcn8s_300.caffemodel           face_seg_fcn8s_deploy.prototxt
3dmm_cnn_resnet_101.zip                 face_seg_fcn8s_300_deploy.prototxt      face_seg_fcn8s.zip
$ sudo cp /face_swap/data/* /usr/local/face_swap/data/ -r

face_swap_image --cfg /usr/local/face_swap/bin/face_swap_image.cfg --input dao.jpg --input min1.jpg --output o2.jpg

Leave a Comment

Configure Selenium and Chrome to use Tor proxy

Published 2018/11/29 by Daoyuan Li

I’ve been trying to configure Selenium and Chrome to use Tor as proxy and constantly getting error messages like the following:

WebDriverException: Message: invalid argument: cannot parse capability: proxy
from invalid argument: Specifying ‘socksProxy’ requires an integer for ‘socksVersion’
(Driver info: chromedriver=2.44.609545 (c2f88692e98ce7233d2df7c724465ecacfe74df5),platform=Mac OS X 10.14.0 x86_64)

In the end I have to use HTTP proxy instead of SOCKS.

Install and start Tor:

brew install tor
brew services start tor

Install privoxy:

brew install privoxy

Configure privoxy (vi /usr/local/etc/privoxy/config) to chain it with Tor:

forward-socks5t   /               127.0.0.1:9050 .

Start privoxy by default on port 8118:

brew services start privoxy

Check if your traffic is proxied:

from selenium.webdriver.common.proxy import Proxy, ProxyType
from selenium import webdriver

prox = Proxy()
prox.proxy_type = ProxyType.MANUAL
prox.http_proxy = "http://localhost:8118"
prox.ssl_proxy = "http://localhost:8118"

capabilities = webdriver.DesiredCapabilities.CHROME
prox.add_to_capabilities(capabilities)


options = webdriver.ChromeOptions()
options.binary_location = '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
# options.add_argument('headless')
# set the window size
options.add_argument('window-size=1881x1280')

# initialize the driver
driver = webdriver.Chrome(options=options, desired_capabilities=capabilities)

driver.get('http://httpbin.org/ip')

Now you should see a different IP than your real one.

Leave a Comment

TeXLive install all packages using command line tlmgr

Published 2017/09/17 by Daoyuan Li

sudo tlmgr install scheme-full

See: https://www.tug.org/texlive/pkginstall.html

Leave a Comment

Docker Postgres “PANIC: could not locate a valid checkpoint record”

Published 2017/09/14 by Daoyuan Li

It seemed that my Postgres database was not properly shut down when rebooting and when I tried to use docker-compose to start it again, the following message was shown in `docker logs`:

PANIC:  could not locate a valid checkpoint record
LOG:  startup process (PID 23) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure
LOG:  database system is shut down
LOG:  database system was interrupted; last known up at 2017-09-14 08:22:04 UTC
LOG:  unexpected pageaddr B/68B26000 in log segment 000000010000000B0000006D, offset 11689984
LOG:  invalid primary checkpoint record
LOG:  unexpected pageaddr B/688F2000 in log segment 000000010000000B0000006D, offset 9379840
LOG:  invalid secondary checkpoint record

To fix this, first shut down this container (docker-compose down), then start the container in interactive mode:

daoyuan.li:~/Projects/magic/stock$ docker run -it -v /Users/daoyuan.li/Projects/magic/postgres_data:/var/lib/postgres/data postgres:9.6 /bin/bash
root@c4d2fb7edcea:/# gosu postgres pg_resetxlog -f /var/lib/postgres/data
Transaction log reset
root@c4d2fb7edcea:/# exit

After the transaction log is reset, everything should be fine. Now you can start your containers again (docker-compose up -d).

Leave a Comment

Be careful with market orders

Published 2017/09/12 by Daoyuan Li

I was testing my algorithmic trading program just now, and experienced an very important issue with market orders.

TL;DR DO NOT USE MARKET ORDERS UNLESS ABSOLUTELY CONFIDENT!!!

Market orders ensure immediate execution, without guarantee of the price of the order. As a result, your order may be executed with a much higher price than you’ve expected, especially when the trading volume is low and the spread is large. In my case, I ended up paying +5% more than the price I’m willing to pay…

Lessons should be learned.

Leave a Comment

Exporting and Importing Elasticsearch Indicies

Published 2016/02/17 by Daoyuan Li

In my project I need to run some local tests with data from a production elasticsearch cluster, so I exported data from the production server and imported to my local cluster. This can also be used when backing up and restoring data. Here’re the instructions.

Before you start, check out the official documentation: Snapshot and Restore.

Backing up/exporting data:

Modify your eleasticsearch configuration file (normally elasticsearch.yml) and add a path.repo line, for example:
```
path.repo: /usr/local/var/backups/
```
Make sure this path has the correct permissions so that elasticsearch can read and write.

Create snapshot:

curl -XPUT http://localhost:9200/_snapshot/my_backup -d '{"type": "fs", "settings": {"compress": "true", "location": "/usr/local/var/backups/"}}}'
curl -XPUT http://localhost:9200/_snapshot/my_backup/snapshot_1?wait_forcompletion=true

Copy the files in the configured location to your local machine.

Restoring/importing data:

Modify your local elasticsearch configuration similarly like step 1 when backing up.
Place the snapshot files to the repo path.

Close your indices:

curl -XPOST http://localhost:9200/knx-bus/_close

Import data:

curl -XPOST http://localhost:9200/_snapshot/my_backup/snapshot_1/_restore?pretty

Reopen your indices:

curl -XPOST http://localhost:9200/knx-bus/_open

It is important that your the elasticsearch version on your importing party is compatible with the one exporting data, i.e., in this case your local machine has to be the same version or newer. If not, you need to upgrade elasticsearch first. The official documentation says:

The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore it’s possible to restore a snapshot made from one cluster into another cluster. All that is required is registering the repository containing the snapshot in the new cluster and starting the restore process. The new cluster doesn’t have to have the same size or topology. However, the version of the new cluster should be the same or newer than the cluster that was used to create the snapshot.

2 Comments

Installing Theano and CUDA on Mac OS X

Published 2015/10/28 by Daoyuan Li

I started trying Theano today and wanted to use the GPU (NVIDIA GeForce GT 750M 2048 MB) on my Mac. Here’s a brief instruction on how to use the GPU on Mac, largely following the instructions from http://deeplearning.net/software/theano/install.html#mac-os.

Install Theano:

$ pip install Theano

Download and install CUDA: https://developer.nvidia.com/cuda-downloads

Put the following lines into your ~/.bash_profile:

# Theano and CUDA
PATH="/Developer/NVIDIA/CUDA-7.5/bin/:$PATH"
export LD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-7.5/lib/
export CUDA_ROOT=/Developer/NVIDIA/CUDA-7.5/
export THEANO_FLAGS='mode=FAST_RUN,device=gpu,floatX=float32'

Note that the PATH line is necessary. Otherwise you may see the following message:

ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.

Configure Theano:

$ cat .theanorc 
[gcc]
cxxflags = -L/usr/local/lib -L/Developer/NVIDIA/CUDA-7.5/lib/

Test if GPU is used:

$ cat check.py 
from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in xrange(iters):
    r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
    print('Used the cpu')
else:
    print('Used the gpu')

$ THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32 time python check.py 
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 1.743682 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu
        2.47 real         2.19 user         0.27 sys
$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 time python check.py 
Using gpu device 0: GeForce GT 750M
[GpuElemwise{exp,no_inplace}(<CudaNdarrayType(float32, vector)>), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 1.186971 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu
        2.09 real         1.59 user         0.41 sys

A more realistic example:

$ cat lr.py 
import numpy
import theano
import theano.tensor as T
rng = numpy.random

N = 400
feats = 784
D = (rng.randn(N, feats).astype(theano.config.floatX),
rng.randint(size=N,low=0, high=2).astype(theano.config.floatX))
training_steps = 10000

# Declare Theano symbolic variables
x = T.matrix("x")
y = T.vector("y")
w = theano.shared(rng.randn(feats).astype(theano.config.floatX), name="w")
b = theano.shared(numpy.asarray(0., dtype=theano.config.floatX), name="b")
x.tag.test_value = D[0]
y.tag.test_value = D[1]

# Construct Theano expression graph
p_1 = 1 / (1 + T.exp(-T.dot(x, w)-b)) # Probability of having a one
prediction = p_1 > 0.5 # The prediction that is done: 0 or 1
xent = -y*T.log(p_1) - (1-y)*T.log(1-p_1) # Cross-entropy
cost = xent.mean() + 0.01*(w**2).sum() # The cost to optimize
gw,gb = T.grad(cost, [w,b])

# Compile expressions to functions
train = theano.function(
            inputs=[x,y],
            outputs=[prediction, xent],
            updates=[(w, w-0.01*gw), (b, b-0.01*gb)],
            name = "train")
predict = theano.function(inputs=[x], outputs=prediction,
            name = "predict")

if any([x.op.__class__.__name__ in ['Gemv', 'CGemv', 'Gemm', 'CGemm'] for x in
        train.maker.fgraph.toposort()]):
    print('Used the cpu')
elif any([x.op.__class__.__name__ in ['GpuGemm', 'GpuGemv'] for x in
          train.maker.fgraph.toposort()]):
    print('Used the gpu')
else:
    print('ERROR, not able to tell if theano used the cpu or the gpu')
    print(train.maker.fgraph.toposort())

for i in range(training_steps):
    pred, err = train(D[0], D[1])

print("target values for D")
print(D[1])

print("prediction on D")
print(predict(D[0]))
$ THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32 time python lr.py 
Used the cpu
target values for D
[ 1.  1.  0.  1.  0.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  0.  0.  1.
  1.  0.  0.  1.  0.  0.  1.  1.  0.  1.  1.  1.  1.  0.  1.  1.  0.  1.
  0.  0.  0.  0.  0.  1.  0.  0.  0.  1.  1.  0.  1.  1.  1.  0.  1.  0.
  0.  0.  0.  0.  0.  1.  0.  1.  0.  0.  0.  1.  1.  1.  0.  0.  1.  1.
  1.  1.  0.  0.  0.  1.  0.  0.  1.  1.  0.  0.  1.  1.  1.  1.  0.  1.
  0.  0.  0.  0.  1.  0.  0.  1.  1.  1.  0.  0.  1.  1.  1.  1.  1.  1.
  1.  1.  1.  1.  0.  1.  1.  0.  0.  1.  0.  0.  0.  1.  0.  1.  1.  1.
  1.  0.  0.  1.  0.  1.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.  1.  0.
  1.  0.  1.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.
  1.  0.  1.  0.  0.  1.  0.  0.  1.  1.  1.  1.  0.  1.  0.  0.  1.  0.
  0.  0.  1.  1.  1.  1.  1.  1.  1.  0.  1.  1.  1.  0.  1.  0.  1.  0.
  0.  1.  1.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  1.  0.  1.  0.  1.
  1.  0.  1.  1.  1.  0.  0.  1.  1.  1.  1.  0.  0.  0.  1.  1.  0.  0.
  1.  0.  0.  0.  0.  1.  1.  1.  0.  1.  1.  1.  0.  1.  0.  0.  0.  0.
  0.  1.  1.  1.  1.  1.  1.  0.  0.  1.  1.  1.  0.  1.  0.  1.  0.  1.
  1.  0.  0.  0.  1.  1.  0.  0.  1.  0.  0.  0.  0.  1.  0.  0.  0.  1.
  0.  1.  0.  1.  1.  0.  1.  1.  0.  0.  0.  0.  1.  0.  0.  0.  0.  1.
  0.  1.  0.  0.  1.  1.  0.  0.  1.  1.  0.  1.  0.  1.  0.  0.  1.  1.
  0.  1.  1.  0.  0.  1.  1.  0.  0.  1.  0.  1.  1.  0.  0.  0.  1.  0.
  0.  0.  1.  0.  0.  0.  0.  1.  1.  0.  1.  1.  1.  0.  1.  1.  1.  1.
  1.  0.  0.  1.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  0.  1.  1.  1.
  0.  1.  1.  1.  0.  0.  0.  0.  1.  1.  1.  0.  0.  0.  0.  1.  0.  0.
  1.  1.  0.  1.]
prediction on D
[1 1 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 0 1 0 0 1 1 0 1 1 1 1 0 1 1 0 1 0
 0 0 0 0 1 0 0 0 1 1 0 1 1 1 0 1 0 0 0 0 0 0 1 0 1 0 0 0 1 1 1 0 0 1 1 1 1
 0 0 0 1 0 0 1 1 0 0 1 1 1 1 0 1 0 0 0 0 1 0 0 1 1 1 0 0 1 1 1 1 1 1 1 1 1
 1 0 1 1 0 0 1 0 0 0 1 0 1 1 1 1 0 0 1 0 1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1
 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 1 1 1 1 0 1 0 0 1 0 0 0 1 1 1
 1 1 1 1 0 1 1 1 0 1 0 1 0 0 1 1 0 0 1 0 0 0 0 0 0 0 1 0 1 0 1 1 0 1 1 1 0
 0 1 1 1 1 0 0 0 1 1 0 0 1 0 0 0 0 1 1 1 0 1 1 1 0 1 0 0 0 0 0 1 1 1 1 1 1
 0 0 1 1 1 0 1 0 1 0 1 1 0 0 0 1 1 0 0 1 0 0 0 0 1 0 0 0 1 0 1 0 1 1 0 1 1
 0 0 0 0 1 0 0 0 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 0 0
 1 0 1 1 0 0 0 1 0 0 0 1 0 0 0 0 1 1 0 1 1 1 0 1 1 1 1 1 0 0 1 0 0 0 0 1 1
 0 0 0 0 0 1 1 1 0 1 1 1 0 0 0 0 1 1 1 0 0 0 0 1 0 0 1 1 0 1]
        8.92 real         8.24 user         1.14 sys
$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 time python lr.py 
Using gpu device 0: GeForce GT 750M
Used the gpu
target values for D
[ 1.  0.  0.  0.  0.  1.  0.  0.  1.  1.  0.  0.  1.  1.  0.  0.  1.  1.
  0.  0.  0.  1.  1.  0.  1.  1.  1.  0.  0.  1.  1.  1.  1.  1.  1.  0.
  0.  1.  0.  0.  1.  1.  0.  0.  1.  1.  0.  1.  0.  1.  1.  0.  1.  1.
  1.  0.  1.  1.  0.  0.  0.  1.  1.  1.  1.  1.  0.  0.  1.  1.  0.  1.
  1.  1.  1.  0.  1.  1.  0.  1.  1.  1.  0.  0.  0.  1.  1.  0.  0.  0.
  1.  0.  1.  0.  0.  0.  0.  1.  1.  1.  1.  0.  0.  1.  0.  1.  0.  1.
  1.  0.  1.  1.  0.  0.  0.  0.  1.  0.  0.  1.  0.  0.  0.  1.  0.  1.
  1.  1.  0.  0.  0.  1.  0.  1.  0.  1.  0.  1.  1.  1.  1.  1.  0.  1.
  1.  0.  1.  1.  0.  0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  0.
  1.  0.  0.  1.  1.  1.  1.  0.  0.  0.  1.  1.  1.  0.  1.  0.  0.  1.
  1.  1.  1.  1.  1.  1.  1.  1.  0.  0.  0.  0.  0.  1.  1.  1.  0.  1.
  0.  1.  0.  1.  1.  1.  1.  0.  0.  0.  1.  1.  1.  1.  0.  0.  0.  1.
  0.  1.  1.  1.  0.  1.  1.  1.  0.  0.  0.  0.  1.  0.  1.  0.  0.  1.
  0.  0.  1.  1.  0.  1.  0.  1.  1.  1.  0.  0.  1.  1.  0.  0.  0.  0.
  1.  0.  0.  1.  0.  0.  0.  0.  1.  0.  0.  1.  1.  1.  1.  1.  1.  1.
  0.  1.  1.  0.  0.  0.  1.  0.  1.  1.  0.  0.  0.  0.  0.  0.  1.  0.
  1.  1.  1.  0.  0.  1.  0.  1.  0.  0.  1.  0.  1.  0.  0.  1.  0.  0.
  1.  1.  0.  1.  1.  1.  0.  0.  0.  0.  0.  1.  0.  1.  0.  0.  0.  1.
  0.  0.  1.  1.  0.  1.  1.  0.  1.  1.  1.  0.  1.  1.  0.  0.  0.  0.
  0.  0.  1.  1.  1.  1.  1.  1.  1.  1.  0.  1.  1.  1.  0.  1.  0.  1.
  1.  1.  0.  1.  1.  0.  1.  1.  1.  0.  0.  1.  1.  0.  0.  0.  0.  0.
  1.  0.  0.  1.  1.  1.  0.  1.  0.  0.  1.  1.  0.  1.  1.  0.  1.  1.
  0.  0.  1.  0.]
prediction on D
[1 0 0 0 0 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 0 1 1 0 1 1 1 0 0 1 1 1 1 1 1 0 0
 1 0 0 1 1 0 0 1 1 0 1 0 1 1 0 1 1 1 0 1 1 0 0 0 1 1 1 1 1 0 0 1 1 0 1 1 1
 1 0 1 1 0 1 1 1 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 1 1 1 1 0 0 1 0 1 0 1 1 0 1
 1 0 0 0 0 1 0 0 1 0 0 0 1 0 1 1 1 0 0 0 1 0 1 0 1 0 1 1 1 1 1 0 1 1 0 1 1
 0 0 1 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 1 1 1 0 1 0 0 1 1 1 1 1 1
 1 1 1 0 0 0 0 0 1 1 1 0 1 0 1 0 1 1 1 1 0 0 0 1 1 1 1 0 0 0 1 0 1 1 1 0 1
 1 1 0 0 0 0 1 0 1 0 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0
 0 1 0 0 1 1 1 1 1 1 1 0 1 1 0 0 0 1 0 1 1 0 0 0 0 0 0 1 0 1 1 1 0 0 1 0 1
 0 0 1 0 1 0 0 1 0 0 1 1 0 1 1 1 0 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 0 1 1 0 1
 1 1 0 1 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 1 1 1 0 1 0 1 1 1 0 1 1 0 1 1 1 0
 0 1 1 0 0 0 0 0 1 0 0 1 1 1 0 1 0 0 1 1 0 1 1 0 1 1 0 0 1 0]
       19.78 real        17.61 user         1.24 sys

So it seems this GPU does not outperform the CPU. Well,GT 750M may not be the best GPU you can get… Someone else here has a similar experience.

5 Comments

Migrating/Installing Prestashop to Amazon AWS EC2

Published 2015/02/25 by Daoyuan Li

Recently I migrated a Prestashop website to AWS/EC2 and the process is quite smooth. Here’s how to do it.

1. Get your Website data: including Prestashop website files and a latest database dump. Remove all the files except index.php in /cache/smarty/compile and /cache/smarty/cache. (See here.)

2. Launch your EC2 instance. But before doing that, check first if a pre-configured or managed Prestashop is available in your preferred region (more info here). If not, choose the appropriate instance type and get it running. In my case I chose Ubuntu Linux.

3. Install the AMP (Apache, MySQL and PHP). On Ubuntu you can simply do it with a few simple commands:

sudo apt-get update
sudo apt-get upgrade
sudo tasksel install lamp-server
sudo apt-get install phpmyadmin

Make sure that you remember your password for MySQL root user and phpmyadmin user.

4. Copy your database dump and website files to your EC2 instance using scp, for instance:

scp -i you-key.pem files.tar.gz [email protected]

5. Create a MySQL user for Prestashop, you don’t want to use the root user for this purpose. You can do this using phpmyadmin or a command line. Also, create a database for Prestashop and grant the user you’ve created just now all privileges on this database.

6. Import your database from the dump:

mysql -u root -p ps < db_dump.sql

7. Extract your website files on your EC2 instance. In my case, inside /var/www/html/. If there is an index.html file inside this directory, remove it first. Make sure you set the correct user and group to all the website files:

sudo chown -R www-data:www-data *
sudo chmod -R 755 *

Note that if don’t set the correct permissions, you may get complaints from Apache, like the following:

PHP Fatal error: Uncaught exception ‘SmartyException’ with message ‘unable to write file /var/www/html/cache/smarty/compile/46/d9/69/wrt54ecd6c1856a72.61943173’ in /var/www/html/tools/smarty/sysplugins/smarty_internal_write_file.php:44

Stack trace:

#0 /var/www/html/tools/smarty/sysplugins/smarty_internal_template.php(201): Smarty_Internal_Write_File::writeFile(‘/var/www/html/c…’, ‘<?php /* Smarty…’, Object(Smarty))

#1 /var/www/html/tools/smarty/sysplugins/smarty_internal_templatebase.php(155): Smarty_Internal_Template->compileTemplateSource()

#2 /var/www/html/classes/controller/AdminController.php(1936): Smarty_Internal_TemplateBase->fetch(‘controllers/mod…’)

#3 /var/www/html/classes/controller/AdminController.php(2337): AdminControllerCore->initModal()

#4 /var/www/html/classes/controller/Controller.php(163): AdminControllerCore->init()

#5 /var/www/html/classes/Dispatcher.php(373): ControllerCore->run()

#6 /var/www/html/admin123/index.php(54): DispatcherCore->dispatch()

#7 {main}\n thrown in /var/www/html/tools/smarty/sysplugins/smarty_internal_write_file.php on line 44

8. Enable mcrypt for PHP:

sudo updatedb
locate mcrypt.ini
> /etc/php5/mods-available/mcrypt.ini
locate mcrypt.so
> /usr/lib/libmcrypt.so.4
> /usr/lib/libmcrypt.so.4.4.8
> /usr/lib/php5/20121212/mcrypt.so
sudo vi /etc/php5/mods-available/mcrypt.ini
sudo php5enmod mcrypt
ls -al /etc/php5/cli/conf.d/20-mcrypt.ini
> lrwxrwxrwx 1 root root 31 Feb 24 19:32 /etc/php5/cli/conf.d/20-mcrypt.ini -> ../../mods-available/mcrypt.ini
ls -al /etc/php5/apache2/conf.d/20-mcrypt.ini
> lrwxrwxrwx 1 root root 31 Feb 24 19:32 /etc/php5/apache2/conf.d/20-mcrypt.ini -> ../../mods-available/mcrypt.ini
service apache2 restart

9. Update your Prestashop settings in the config/settings.inc.php file. Set the correct database information.

10. Enable mod_rewrite:

sudo a2enmod rewrite
sudo service apache2 restart

Also, make sure to add something like the following in your apache site configuration file:

        <Directory />
                Options FollowSymLinks
                AllowOverride all
        </Directory>
        <Directory /var/www/html/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride all
                Order allow,deny
                allow from all
        </Directory>

And then remember to restart your apache server.

11. Install sendmail:

sudo apt-get install sendmail

12. Go to your Prestashop back office and reconfigure your domain. Go to Preferences -> SEO & URLs and scroll down to Set shop URL section.

Now everything should be OK, your new Prestashop website is available on your new domain!

If you have any issues, just drop a line here or email me.

4 Comments

OSX Yosemite cannot boot when TRIM is enabled on third-party SSD

Published 2014/10/18 by Daoyuan Li

After upgrading my MBP to OSX Yosemite and enabling TRIM, the computer cannot boot into the system after restart. It stuck with a no-entry sign.

You can solve this issue by following instructions from http://www.cindori.org/trim-in-os-x-yosemite/

Boot your Mac in Recovery Mode by holding Cmd-R during boot

Open the Terminal from the menu bar

Run these commands, replacing YourDisk with the name of your Mac disk

rm -rf /Volumes/YourDisk/System/Library/Extensions/IOAHCIFamily.kext
cp -r /System/Library/Extensions/IOAHCIFamily.kext /Volumes/YourDisk/System/Library/Extensions/IOAHCIFamily.kext
touch /Volumes/YourDisk/System/Library/Extensions
kextcache -u /Volumes/YourDisk

For me, I have to first use the disk utility to unlock my SSD first since I have Filevault enabled, otherwise the first command above will complain about “readonly” files.

Since TRIM is very helpful for SSDs, it’s encouraged to have it enabled for better performance. You can use Trim Enabler v3.2.5 and above without issues. But note that it disables kext signing on your mac, “it still leaves you with the same amount of security as in OS X Mavericks, where the kext signing requirement didn’t exist”. (http://www.cindori.org/update-on-trim-in-yosemite/)

4 Comments

WordPress pages load very slowly after Google is blocked

Published 2014/06/18 by Daoyuan Li

After Google is completely blocked in China since May 31, I’ve had problems loading my website. After some investigation I found out the problem lies in loading failure of Google Fonts. Since it takes a long time to fail, the page waits before it fails and display with the fallback styles.

To solve the problem, I crawled the font styles from Google and put it up on my server. Then I changed the WordPress theme to use the font style on the server. Everything seems to work perfectly after that.

Another option would be completely disable Google Font Styles. This can be easily done using the WP plugin “Disable Google Fonts”, if you’re using WP’s default twenty-* themes. In my case, I would have to comment out two lines in Mog Theem Functions file (functions.php). Since my theme relies on Google Fonts, instead of commenting out the lines, I changed these to lines to use the localized style file:

	//wp_enqueue_style( 'opensans', 'http://fonts.googleapis.com/css?family=Open+Sans');
	//wp_enqueue_style( 'josefinslab', 'http://fonts.googleapis.com/css?family=Josefin+Slab:600');
	wp_enqueue_style( 'opensans', '/gfonts.css');
	wp_enqueue_style( 'josefinslab', '/gfonts.css');

Leave a Comment

Raspberry Pi as a toy web server

Published 2014/04/24 by Daoyuan Li

1. Tell my router to route certain traffic to the Raspberry Pi. For example, I route HTTP and SSH traffic to one of my Pi’s. I disabled password login for SSH, using public key authentication instead: in /etc/ssh/sshd_config, use the following setting and restart SSH service using `sudo service ssh restart’.

RSAAuthentication yes
PubkeyAuthentication yes

# To disable password authentication:
ChallengeResponseAuthentication no
PasswordAuthentication no
UsePAM no

# To disable root login:
PermitRootLogin no

2. Getting my IP address of the Raspberry Pi. I firstly created a PHP script on my domain to record the IP address in a text file.

<?php                                                                            
                                                                                 
$token = 'secret';                   
                                                                                 
if ($_GET['token'] == $token) {                                                  
    if (!empty($_SERVER['HTTP_CLIENT_IP'])) {                                    
        $ip = $_SERVER['HTTP_CLIENT_IP'];                                        
    } elseif (!empty($_SERVER['HTTP_X_FORWARDED_FOR'])) {                        
        $ip = $_SERVER['HTTP_X_FORWARDED_FOR'];                                  
    } else {                                                                     
        $ip = $_SERVER['REMOTE_ADDR'];                                           
    }                                                                            
    $fp = fopen('home_ip.txt', 'w');                                             
    fwrite($fp, $ip);                                                            
}                                                                                
                                                                                 
?>

Then tell the Raspberry Pi to report its IP address every 5 minutes, using crontab:

pi@alice ~ $ crontab -l
*/5 * * * * curl daoyuan.li/home_ip.php?token=secret

After a while the IP address is recorded in the text file and updated every 5 minutes.

3. Optionally create a DNS record for the Pi. I use Cloudflare to manage DNS settings by myself, so just add/update an entry in Cloudflare’s settings. I point pi.daoyuan.li to the IP address of one Pi. This can be done automatically in the future.

4. Install Flask on the Pi.

sudo apt-get update
sudo apt-get install python-pip
sudo pip install Flask

5. Install nginx and uwsgi on the Pi.

sudo apt-get install nginx
sudo apt-get install build-essential python python-dev
sudo apt-get install python-virtualenv
sudo pip install uwsgi

6. Set up nginx along with uwsgi and Flask.

mkdir flask && cd $_
virtualenv env
. env/bin/activate
pip install Flask

Edit nginx config:

~/flask $ cat flask_nginx.conf 
server {
    listen      5000;
    server_name localhost;
    charset     utf-8;
    client_max_body_size 75M;

    location / { try_files $uri @flask; }
    location @flask{
        include uwsgi_params;
        uwsgi_pass unix:/home/pi/flask/uwsgi.sock;
    }
}

sudo rm /etc/nginx/sites-enabled/default
sudo ln -s flask_nginx.conf /etc/nginx/sites-enabled/default 
sudo service nginx restart

Edit uwsgi config:

~/flask $ cat flask_uwsgi.ini 
[uwsgi]
#application's base folder
base = /home/pi/flask

#python module to import
app = hello
module = %(app)

home = %(base)/env
pythonpath = %(base)

#socket file's location
socket = /home/pi/flask/uwsgi.sock

#permissions for the socket file
chmod-socket    = 666

#the variable that holds a flask application inside the module imported at line #6
callable = app

#location of log files
logto = /home/pi/flask/uwsgi.log

Create a simple Flask app:

~/flask $ cat hello.py
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

Start up uwsgi:

uwsgi --ini flask_uwsgi.ini &

7. Done! http://pi.daoyuan.li:5000/

Update on June 24, 2014:

Getting the external address in step 2 can be done by running this command in Raspberry Pi:

curl ifconfig.me

See: http://www.commandlinefu.com/commands/view/5427/get-your-external-ip-address

Leave a Comment