Up until today (April 20, 2025) I’ve hosted this WordPress site using a variety of services and software. This site originally started out on Google Cloud Platform but then slowly transitioned to self-hosting via Raspberry Pi 4. For reasons that I do not remember, when migrating over to rpi4 I set up the site and all its components directly in the host server, without using any containerization software such as Docker. Retroactively speaking, I likely did it to brush up on my linux knowledge and make sure that I understood everything that was happening under the hood of my server and website. I’ve created and documented my journey through the various migrations in this site but finally realized recently that it is time to move on to using new tools and services. To “catch up with the times” or so people say. Now that I spent enough time building code repos from source, manually orchestrating the deployments and upgrades of WordPress and MariaDB, and manually updating PHP versions by hand, I will now leave that to the community and embrace the wonderfulness that is waiting for new software versions to arrive in docker containers 😉

About Caddy and FrankenPHP

Apache2 was (and likely still is) the go-to webserver for WordPress deployments. For WordPress users, transitioning from the traditional Apache2 setup to a more modern and efficient architecture can yield significant benefits. Caddy, with its V2 released in 2020, is a new webserver that, though young, promises high performance with an easier to use configuration. FrankenPHP is a cutting-edge PHP application server built on the Caddy web server. It consolidates the web server and PHP runtime into a single service, eliminating the need for separate PHP-FPM and Apache processes. This integration simplifies deployment and enhances performance. Notably, benchmarks have shown that FrankenPHP can process requests significantly faster than traditional setups .​

Current System Setup

• WordPress v6.7
• Apache/2.4.62
• FPM-PHP (PHP v8.2.28)
• MariaDB 10.5.26

Prerequisites

• Backup the site using your WP site migration tool of choice (I used UpdraftPlus’s Free Version)
• Backup your MySQL/MariaDB as well – if you decide to upgrade your DB version then you may potentially come across compatibility issues, and your database tables may need to be repaired in order for it to be useable
• Docker and Docker-Compose. Below is the version I’m using at the time of writing this post:
  Docker version 28.0.4, build b8034c0
Docker Compose version v2.34.0
• You will need to retrieve all of the username/passwords for the following
  • WordPress DB name
  • WordPress DB username/password
  • MariaDB root password

How to Migrate

Note: There will be site downtime while you perform this migration using this manual.

Creating the Docker Compose Configuration File

I will be using FrankenPHP’s officially (yet not official at the same time…) sponsored WordPress images created from the FrankenWP project. The list of latest images can be found here. For MySQL/MariaDB, choose the version that is closest if not equal to the version you are currently using – if you perform too big of a version jump the built-in upgrade feature may not work for your existing data. Lastly, if you use PHPMyAdmin, also grab and configure the details for that. Long story short, Caddy nor FrankenWP official pages had anything close to a work-able compose.yaml, and a lot of trial-and-error and extra configurations were necessary in order to at least properly serve the site in Production.

Below is the compose.yaml file I finally ended up with that made the whole thing work, though I’m still giving most credit to FrankenWP’s creator since I took the template from the project’s examples section. Note that the below example has sample data (denoted by ##) which will need to be replaced with info from your current WP site and DB info:

services:
  wordpress:
    image: wpeverywhere/frankenwp:latest-php8.3
    restart: always
    ports:
      - "80:80" # HTTP
      - "443:443" # HTTPS
    environment:
      SERVER_NAME: ${SERVER_NAME:-:80}
      WORDPRESS_DB_HOST: ${DB_HOST:-db}
      WORDPRESS_DB_USER: ${DB_USER:-wordpress}
      WORDPRESS_DB_PASSWORD: ${DB_PASSWORD:-##YOUR_WP_DB_PASSWORD##}
      WORDPRESS_DB_NAME: ${DB_NAME:-wordpress}
      # Currently WP_DEBUG turns on even if below is set false. To turn debug off, remove it entirely.
      # See:  https://github.com/docker-library/wordpress/issues/496
      #WORDPRESS_DEBUG: false
      WORDPRESS_TABLE_PREFIX: ${DB_TABLE_PREFIX:-wp_}
      CACHE_LOC: ${CACHE_LOC:-/var/www/html/wp-content/cache}
      TTL: ${TTL:-80000}
      PURGE_PATH: ${PURGE_PATH:-/__cache/purge}
      PURGE_KEY: ${PURGE_KEY:-}
      BYPASS_HOME: ${BYPASS_HOME:-false}
      BYPASS_PATH_PREFIXES: ${BYPASS_PATH_PREFIXES:-/wp-admin,/wp-content,/wp-includes,/wp-json,/feed}
      CACHE_RESPONSE_CODES: ${CACHE_RESPONSE_CODES:-000}
        # Enable debug to check for Caddy issues (Certificate renewal error, config error, etc)
      #CADDY_GLOBAL_OPTIONS: |
        #email ##EMAIL_ADDRESS##
        #auto_https disable_redirects
        #debug
      WORDPRESS_CONFIG_EXTRA: |
        # Add extra configs here as necessary
        define('WP_SITEURL', '##SITEURL##');
        define('WP_HOME', '##WP_HOME##');

    volumes:
      # Mount your WordPress content directory if needed
      - /var/www/html/wp-content:/var/www/html/wp-content
      # Load your own Caddyfile to configure site
      - ./Caddyfile:/etc/caddy/Caddyfile
      # Store SSL Certs so that it can be reused when containers are recreated
      # (and do avoid exceeding cert issue rate limits when debugging containers)
      - ./caddy_data:/data
      # Other root files, such as ads.txt
      - /var/www/html/ads.txt:/var/www/html/ads.txt

    depends_on:
      - db
    tty: true
    networks:
      wpnet:
        ipv4_address: 172.28.0.3

    # Below only if necessary
    #extra_hosts:
    #  - ##TLD##:172.28.0.3
    #  - www.##TLD##:172.28.0.3

  db:
    # Be careful of making too big of a version jump...
    # Slowly update db version to reduce chance of data corruption
    image: mariadb:11.7.2
    restart: always
    ports:
      - 3306:3306
    environment:
      MYSQL_DATABASE: ${DB_NAME:-wordpress}
      MYSQL_USER: ${DB_USER:-wordpress}
      MYSQL_PASSWORD: ${DB_PASSWORD:-##YOUR_WP_DB_PASSWORD##}
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-##YOUR_ROOT_PASSWORD##}
    volumes:
      - /var/lib/mysql:/var/lib/mysql
      - /etc/mysql/mariadb.conf.d:/etc/mysql/mariadb.conf.d
    networks:
      wpnet:
        ipv4_address: 172.28.0.4

  phpmyadmin:
    # using arm64 version since I am on raspberry pi
    image: arm64v8/phpmyadmin:latest
    restart: always
    ports:
      - ${LOCAL_PHPMYADMIN_PORT:-8086}:80
    environment:
      PMA_HOST: db
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-##YOUR_ROOT_PASSWORD##}
    depends_on:
      - db
    networks:
      wpnet:
        ipv4_address: 172.28.0.5
    volumes:
      # Use existing phpmyadmin data
      - /var/www/html/phpmyadmin:/var/www/html/phpmyadmin

networks:
  wpnet:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16
          gateway: 172.28.0.1

Mix and match whatever you need from the above to get your own site working. Below is a list of some configurations which can be made differently based on your unique situation:

• Remove PHPMyAdmin if you don’t use it.
• Above config uses a bridge network. This allows for multiple sites to be hosted on the same host device via reverse proxy. Also, as you will see later, if your PHPMyAdmin also uses standard HTTP/S ports then it is not possible to run both your site (WordPress) and PHPMyAdmin on the same port in the host network configuration.
• Static addresses can be used to grant read/write privileges for the DB which is hosted in another instance and by default only allows for localhost connection. Static IP addresses can be omitted if you configure your MySQL/MariaDB to allow connections from the entire IP address range in your bridge network (example further below). It can also be omitted if you decide to host your containers using the host network configuration.
• Static IP address of PHPMyAdmin is also used in the below Caddyfile to set up reverse proxy. There’s probably a better way to do this without static IP addresses… will update later…

Creating the Caddyfile

Now that our docker configurations are set up, we need to configure Caddyfile. This Caddyfile should be placed in the same folder as your compose.yaml:

{
        # Uncomment below when experiencing redirect loops from reverse proxy/cloud load balancer/etc
        #auto_https disable_redirects
        {$CADDY_GLOBAL_OPTIONS}

        frankenphp {
                #worker /path/to/your/worker.php
                {$FRANKENPHP_CONFIG}
        }

        # https://caddyserver.com/docs/caddyfile/directives#sorting-algorithm
        order php_server before file_server
        order php before file_server
        order wp_cache before rewrite
        order request_header before wp_cache
}

{$CADDY_EXTRA_CONFIG}

##WWW_DOMAIN## {

        tls ##EMAIL_ADDRESS##

        @phpmyadmin path /phpmyadmin*
        reverse_proxy @phpmyadmin 172.28.0.5:80

        @static {
                file
                path *.ico *.css *.js *.gif *.jpg *.jpeg *.png *.svg *.woff *.txt
        }

        root * /var/www/html/
        encode br zstd gzip

        wp_cache {
                loc {$CACHE_LOC:/var/www/html/wp-content/cache}
                cache_response_codes {$CACHE_RESPONSE_CODES:2XX,404,405}
                ttl {$TTL:6000}
                purge_path {$PURGE_PATH:/__cache/purge}
                purge_key {$PURGE_KEY}
                bypass_home {$BYPASS_HOME:false}
                bypass_path_prefixes {$BYPASS_PATH_PREFIXES:/wp-admin,/wp-json}
        }

        {$CADDY_SERVER_EXTRA_DIRECTIVES}
        php_server
}

# For www redirects
###DOMAIN## {
#        redir https://www.{host}{uri} permanent
#}

Replace the ##ITEM## with configurations for your own site. At the very bottom is a redirection bit in case you want www to be redirected to your TLD+1, or vice versa. The PHPMyAdmin reverse proxy is also configured inside this Caddyfile.

Execute the Migration

Now it’s time to make the transition. Bring your webserver and DB down from your host server and start your docker configurations. If at any point in this step something goes wrong, you can revert to your existing host server setup by reversing the commands below (and in the reverse execution order):

# FYI if you do not stop Apache, you will get the following error:
#Error response from daemon: failed to set up container networking: driver failed programming external
#connectivity on endpoint frankenphp-wordpress-1
#(931a4e79170c448d3aca76617cc7be9847f227138f70a4ef085efea09e19eeb3): failed to bind host port for
#0.0.0.0:80:172.18.0.2:80/tcp: address already in use

# Also need to stop MariaDB, or else you may get corrupt data from multiple live services at the same time.
sudo systemctl stop apache2
sudo systemctl stop mariadb
docker compose up

Once the instances have started up fully, let’s go and make the change in MariaDB to allow both wordpress and phpmyadmin instance access. Below settings incorporate some wildcards in the docker network which can set both wp and phpmyadmin at the same time, but if you don’t use phpmyadmin you can just limit to the WP ip address only:

docker exec -it ##DIR##-db-1 bash
mariadb -u root -p
# enter your root password

# replace each of the data with the items you configured in the compose.yaml
# 172.28.%.% is a wildcard match to all IP addresses in your docker bridge network.
GRANT ALL PRIVILEGES ON ##WORDPRESS_DB##.* TO '##WORDPRESS_USER##'@'172.28.%.%' IDENTIFIED BY '##WORDPRESS_DB_PASSWORD##';
FLUSH PRIVILEGES;

Now your WordPress and PHPMyAdmin should be able to access your DB normally. However, you may need to wait a few minutes for Caddy to use LetsEncrypt to fetch a new certificate for you. You can use docker logs ##DIR##-db-1 -f to live tail your wordpress log to check on SSH certification creation/renewal progress. After a while you should be able to access your site from the public as normal.

Other Notes

Under the above setup, if you use Site Kit by Google, it will need to be reconfigured every time you create a new docker container for wp. This means there’s data used by Site Kit which resides outside of the wp-content directory which we bind to. Will update this guide once I figure out what else is necessary to bind to the docker containers.

Additional Troubleshooting

PHPMyAdmin – mysqli_real_connect(): (HY000/2002): No such file or directory

Change $cfg['Servers'][$i]['host'] value to 127.0.0.1

Unable to Access MySQL or MariaDB

In /etc/mysql/mariadb.conf.d/50-server.cnf, change bind-address to 0.0.0.0

Situation

Below is a roughly visualized grouping of all stakeholders involved in this incident:

• A : Myself, the claimant who will file a claim against the management company (E) for damages and compensation.
• B : My company with which the lease is contracted to (法人契約). In other words, I participate in “company housing”.
• C : The mandatory fire insurance coverage I signed upon leasing this property.
• D : Management company’s 24/7 support center.
• E : Management company. Legally responsible for reducing rent (or equivalent exchange) under Civil Law 611.
• F : Corporate insurance of the management company (D). Compensates me (A) for personal property damage and accommodation during the repairs. Insures management company (D) for the necessary water leak repairs to the property.
• G : Water pipe maintenance dispatched by management company (D) to fix water leak.
• H : Inspection dispatched by management company (D) to assess property damage.

Background

• My apartment is 1DK and approximately 39 square meters.
• The lease is not contracted to me, but to my company, thus a 法人契約.
• Although the leak came from the upper floor tenant, they were not liable as the damages came due to an issue with the building’s water infrastructure.
• I have fire insurance (火災保険) which actually covers many non-fire-related incidents, including the water leak in this article.

Compensation

Since I was 0% at fault for this incident the apartment management company paid for all the repairs, including switching out the wallpaper/flooring/cabinets for new ones. There were several other costs and damages incurred which I will categorize into two items: personal item damages and others.

First, due to the water leakage several of my appliances suffered some damage. This included the Google Nest Hub which I placed on my kitchen countertop, and the microwave oven which is placed above my refridgerator. Water got into the wall plug where I had my Nest Hub plugged in, essentially shorting and breaking it. As for the microwave, while it still functions, water got inside of the electronic circuitry which can result in decreased performance. For these two appliances I was given 10,000JPY and 30,000JPY, respectively, with the logic that replacing these would cost roughly those amounts.

The remaining compensation is non-physical and comprises of the following: “apology money”, and accommodation compensation. The apology money of 30,000JPY was the amount given to me while I was inconvenienced as a result of this water leak. I want to note here that the original amount proposed to me was 20,000JPY but I was able to negotiate for a higher amount – my argument was that the costs for purchasing groceries and cooking by myself over two months would be 500JPY/day x 60 days = 30,000JPY. While this amount may sound small to many people, I had a feeling that they wouldn’t accept to pay me for any higher amount and I just wanted to have this issue resolved ASAP.

Next was the accommodation reimbursement, which the management company agreed to pay 8000JPY per day of repairs. As repairs spanned over 5 days (not counting the weekends when repairs were paused), this amounts comes out to be a total of 40,000JPY. While unfortunately hotel costs went over this amount, the remaining (~50000JPY) could be reimbursed by my personal fire insurance policy, meaning I essentially pay nothing out-of-pocket for staying at hotels which the repairs were underway.

In total, 110,000JPY was reimbursed to me by the management company as a result of this incident. Separately, _____JPY was reimbursed by my personal fire insurance policy.

Timeline

5 Must-Do Items

Understand your rights as a renter

First, check some materials on renter’s rights and basic agreements that are made between the owner and the renter. Here’s a good English guide to get started. If incidents occur regardless of whether the fault if your’s or not, contact your personal property (most likely fire but may also contain others like earthquake, etc) insurance company anyway. Depending on the situation you may be able to use your personal insurance to either get money back or pay less in liability damages.

If the damages get to a point where it’s impossible for the property to provide a reasonable level of living condition, then you may be able to threaten your property owner to reduce your rent as pursuant of the Amended Civil Code 611.

Lastly, be careful when you decide to move out of the property and the inspectors mark the areas fixed as needing replacement. It’s possible that the repairs done for the incident was shoddy and that there are still some issues in some hidden areas. If you feel that the inspection was conducted unfairly, show evidence that repairs have been done on these areas and that any damages were a result of daily wear-and-tear (which by law cannot be deducted from your deposit).

Push and complain to the management company often

Honestly, the old man who was in charge of my case was probably coasting through the rest of his employment years until retirement. Throughout this entire incident I had to constantly call and ask for updates and remind him to do his own job. It was the worst customer service experience I ever had to witness and I would file a 1/10 feedback to the management company if there is an option to do so.

While I had a terrible experience with this particular management company, in general it’s always a good idea to call and complain to them as often as you can or else they will just proceed with the minimum amount they can get away with. Complaining to them often helps to communicate the urgency in your incident and speed up the process, and you may even get a higher amount of compensation as a result (see “Compensation” section above where I got an extra 10000JPY just for negotiating with them).

Take and request evidence of EVERYTHING

Take pictures and videos of the entire incident as it unfolds, and record conversations between all the involved parties. In this case my personal fire insurance company was very helpful and cooperative in getting money from the management company, but it’s possible that relevant parties do not honor the agreements made during phone call, in which having a recording of the conversation can be very helpful (Aside: you are allowed to record conversations without the other party’s consent as long as you don’t broadcast it to a wider audience). Obviously having a written letter of the negotiation agreements mailed to you is highly recommended to avoid confusion – while management companies may not deal with email communication, they normally send you letters to confirm details regarding the incident.

Summary

This entire ordeal took 5 months from start until receiving my compensation. The process may have taken slightly longer than necessary due to end of year/beginning of year holidays, but a lot of the holdup was due to the management company not taking my case seriously. I’m also glad that I was able to use my personal fire insurance and felt that the premiums I had to pay for more than four years finally paid off. Nevertheless, I was fully compensated for my troubles though I hope to never have to go through this kind of situation again…!

Are you tired of the daily city traffic and looking for an eco-friendly and convenient way to get around the city? The Evercross Electric Kickboard might be just what you need. In this review, we’ll dive into the specs, benefits, and drawbacks of the Evercross Electric Kickboard. We’ll also cover the rules and regulations of electric scooter usage in Japan, how to register your electric scooter, and what to keep in mind while riding one.

Specs

Advantages

• Cost-saving and eco-friendly mode of transportation – you can save money from riding the train/bus and may even save time depending on where it is you want to go.
• 25km distance gets you far! For example, the below is a picture showing a 10km radius from Shibuya Station. You can go as far as Ueno or cross the Tamagawa river into Kanagawa Prefecture:

• Speed can go up to 30km/hr, but can also be a disadvantage (see rules and regulations further down the article).
• Lightweight and easy to carry – comes with a bag you can put the kickboard in to carry with you onto the train/bus.

Pitfalls

• Be careful of the brake in steep downhill – though this scooter’s brakes are regenerative brakes, it is not powerful enough to stop you completely in a steep decline. It is advised to control your speed and use both the EBS and foot brake on the back of the scooter in such situations.
• It can take a while to charge – I’ve never charged from 0% battery before but I remember it taking 4-6 hours to charge fully.
• Not to be used in substantially rainy conditions – it can take a light drizzle but water is prone to run into the circuits and battery. I’ve had to call customer support to send a new battery once after using it in heavy rain!

Rules and Regulations in Japan

In Japan, electric scooters like the Evercross Electric Kickboard are becoming increasingly popular due to their eco-friendliness and convenience. However, it’s important to keep in mind the rules and regulations of using electric scooters on Japanese roads. According to the Japanese government, electric scooters can only be used on roads with a speed limit of 30 km/h or less, and riders must have a valid driver’s license when using faster electric scooters such as this one.

With regards to whether this scooter can be used on pedestrian walkways, the official answer is NO. However, at its slowest speed setting the electric kickboard runs up to 15km/hr, which is an acceptable speed to use on pedestrian walkways. It is not clear if regulations look at a kickboard’s maximum speed or the speed the rider happens to be using at that time, so proceed at your own risk.

Another thing you will need to do is to register your electric kickboard at your local ward office. You will be required to register the make/model, when you purchased it, serial number, and the potential maximum speed it can run. In exchange you will receive a license plate which you will need to affix onto your scooter. NOTE: This kickboard wasn’t designed to have a license plate attached to it, so you will need to DIY it!!

Lastly, it is necessary for you to purchase liability insurance (自賠責保険) when operating your electric kickboard. Insurance provider rates differ but not by a significant amount. For electric kickboards you will need to purchase the type used for e-scooters (up to 125cc “原付”). For example, Sompo Japan charges 8850JPY for two years of insurance, but it is up to you how long you would like to use the scooter. Personally I imagine the lifetime of an electric kickboard to be around two years. Technically you can “transfer” the insurance to a new electric kickboard but it is up to you to decide how long you plan to use it.

Personal Anecdotes

I have been using the Evercross Electric Kickboard for about a year now, and I am generally satisfied with my purchase. My battery broke down from heavy rain once but it was within the 6-month warranty so I had them ship another battery to me. Replacing the battery wasn’t trivial but I had the tools to get it done. Not only is it an eco-friendly mode of transportation, but it’s also incredibly convenient and easy to use in Tokyo. There are some areas in Tokyo that aren’t quite convenient to get to by train – the electric kickboard is absolutely perfect in these situations.

Conclusion

In conclusion, if you’re looking for an eco-friendly and convenient way to get around the city, the Evercross Electric Kickboard is a great option. Keep in mind the rules and regulations of electric scooter usage in Japan, and consider your personal needs and preferences before making a purchase. You can easily purchase the Evercross Electric Kickboard here on Amazon (clicking on these links help to support me and this website.. thank you so much!!).

Edit (August 4, 2021) : I have received my PR card. Updating documents required and all sections after and including “During Processing”.

Edit (April 14, 2021) : Added additional documents required for PR submission (self, proxy/agent). Filled in “Submitting Your Documents” section. Fixed typos. My PR application is currently still being processed.

Legal Disclaimer : The contents of this article do not constitute legal advice, are not intended to be a substitute for legal advice and should not be relied upon as such. You should seek legal advice or other professional advice in relation to any particular matters you or your organization may have.

This guide outlines my experiences transitioning from an HSP (HSFP) Visa to Japan’s permanent residency. I have lived in Tokyo, Japan since March 2014, starting out with an Engineer Working Visa before transitioning to HSP Visa (90+ points) in April 2019. One who maintains their 80+ points after a year of holding the HSP visa will be given the opportunity to apply for Japan’s permanent residency.

The permanent residence application process may vary from individual to individual, so please do go through the “Before You Read Further” section to understand my situation and background before going through the prepared documents and process. However, one thing you should know before this is that I worked with Deloitte to process my application, and there may be documents which may NOT actually be necessary but were provided for the sake of fortifying my application or just as a just-in-case. The permanent residency application requires so much of your personal information it will be like taking off all your clothes and showing them your bare naked body… the government will give a VERY thorough screening of your situation and background so it is best to provide them with as much transparency and info as possible to reduce delays to your application.

That’s it for the disclaimer… so let’s get right into it!

Before You Read Further

Below are some additional information and assumptions I am making for the sake of letting the reader understand my situation. Like I mentioned in the preface, my situation may not be applicable to everyone.

• I consulted Deloitte for my PR application.
• I am an American citizen.
• I took the HSP -> PR route instead of the normal method of 10+ years living in Japan.
• I had 80+ points the year I obtained my HSP plus the year afterwards, qualifying me to apply for PR status.
• My Reason Letter (more details below) was translated with Deloitte’s help.
• My guarantor is a close friend and is Japanese (FYI you may choose to have another PR individual be your guarantor).

Please refer to this article plus others around the web for a more complete grasp of the PR process for your specific situation.

Preparation

Required Documents (Self)

For the sake of being concise, this will be in list-form with additional points underneath each for clarification purposes. If I have time I will do my best to elaborate further.

• Passport (Copy)
• Residence Card (Copy)
• Portrait Photo – Picture Booths
  • Taken within three months against white background. Other requirements same as Residence Card photo.
• Point Calculation Table for Highly Skilled Professional Status (Previous Year + Current Year)
  • One table for each year (so two tables in total)
  • Remember to sign both documents
• Certificate of Residence (住民票 / Juminhyo) – Ward Office
  • Taken within three months. Must NOT contain personal numbers (MyNumber, Residence Card Number, etc)
  • Nationality, Visa status, etc can be included
• Employment Certificate
  • Should contain company name, start of employment date, current salary + bonus, and position.
• Tax Withholding Statement (源泉徴収書 / Gensen Choshu Sho)
  • Previous year’s. This was an additional document Immigration requested after my PR application was submitted.
• Residence Tax Certificate (納税証明書 / Nozei Shomeisho) – Ward Office
  • Request for the most recent document on file
• Residence Tax Certificate (課税納税証明書 / Kazei Nozei Shomeisho) – Ward Office
  • Request for the most recent document on file
  • Note that this will look similar to Nozei Shomeisho except with a few unmasked numbers.
• Certificate of Income Tax Payment (国税納税証明書 その他３ – Kokuzei Nozei Shomeisho Type 03) – Tax Office
  • Note that you will need to specify WHICH categories to show. In the request form, fill in the following (manually write it if not included as an option):
    • 源泉所得税及び復興特別所得税
    • 申告所得税及び復興特別所得税
    • 消費税及び地方消費税
    • 相続税
    • 贈与税
  • This document is proof that you don’t have any missing/unpaid taxes. If there is a section where it states you have unpaid taxes, you would have a problem 😐
• Monthly Pension Record (各月の年金記録 / Kakutsuki no Nenkin Kiroku) – Pension Office
  • Apparently Deloitte wanted me to retrieve this information from the online “Nenkin Net” site. You may need to visit the Pension Office in your ward to obtain a registration number to make your online account process smoother.
• Bank Statement(s)
  • Bank name, account holder name, current balance, and timestamp needs to be visible. Apparently online screenshots containing all the info is OK.
• Other documents indicating financial assets, properties in Japan
• Letter of Reason
  • This needs to be in JAPANESE. Legal services can help with translation (for a fee)
  • Remember to sign the document. I’ve signed mine after the header and before the introduction.
  • Essentially, should contain your life before Japan, experiences in Japan, future plans, and why you want PR.
  • Wanting to purchase and own property in Japan is a VALID reason for applying for PR status.

Required Documents (Guarantor)

Please reach out to your potential guarantor and have him/her prepare the documents while you prepare your’s. Both your’s and guarantor’s documents are to be submitted together.

• [If Necessary] Guarantor – Permanent Residence card (Copy)
  • Only if your guarantor is also PR.
• Guarantor – Certificate of Residence (住民票 / Juminhyo)
  • Taken within three months. Must NOT contain personal numbers (MyNumber, Residence Card Number, etc)
• Guarantor – Employment Certificate
  • Should contain company name, start of employment date, current salary + bonus, and position.
• Guarantor – Residence Tax Certificate (納税証明書 / Nozei Shomeisho)
  • Request for the most recent document on file.
• Guarantor – Residence Tax Certificate (課税納税証明書 / Kazei Nozei Shomeisho)
  • Request for the most recent document on file.
  • Note that this will look similar to Nozei Shomeisho except with a few unmasked numbers.
• Guarantor – Letter of Guarantee (身元保証書)
  • Affix seal if Japanese, or sign if PR holder.

Required Documents (Proxy/Agent)

The following are required when using a proxy/agent who will submit the application in your stead. Of course, the proxy or agent should provide these documents for you to sign:

• Proxy/Agent Request Form (依頼書)
• Letter of Power (委任状)

Submitting Your Documents

If you are applying for PR by yourself without using a proxy, once you’ve collected everything from the above it’s time to prepare for submitting the document. You’ll need to fill out additional forms outlined below.

• Application for Permanent Residence (永住許可申請書)
  • Remember to glue your portrait photo to top-left of page 1 and sign on page 2.
  • This should be filled out in JAPANESE, so if you have trouble contact your friendly Japanese-fluent friend for assistance.
• Certificate For Payment of Fee (手数料納付書)
  • 8000JPY

Bring all of the above and your actual passport and residence card (NOT copies) with you to the Immigration Services Agency or Foreign Resident Information Center for submission. Your passport and residence card should be returned to you immediately after verification.

During Processing

Once your documents have been submitted it is just a matter of waiting. However, there may be cases where Immigration will ask for additional documents. In my particular case, Immigration asked for my previous year’s tax withholding statement.

Obtaining Your PR Card

You will receive a notification from your proxy/agency or directly from Immigration if you are approved/rejected the PR.

If using a proxy/agency, mail your passport and current residence card to them and they will retrieve and mail it back to you along with your new permanent residence card. If you are applying by yourself, you may go to Immigration with your passport and current residence card to retrieve your new PR card.

Final Thoughts

Below is the complete timeline of my PR application:

Excluding the time it takes to prepare the PR application, it only took about five months from submission to approval. From other sources it normally takes longer, but I heard from another coworker that they also obtained their PR approval within the same amount of time. Note that many factors (individual qualifications, Covid, etc) may be factored in how long it takes for your own PR to be approved, but expect anywhere between 5 – 12 months for processing.

When you have PR, new options become available for you. Below are some potential perks you are now able to enjoy:

• While your visa is active indefinitely (no time limit), you do need to renew your card every seven years. More info here.
• Without PR banks will likely require you to pay a down payment for housing loans (approximately 10-30%). This becomes 0 if you have PR. In some cases you may negotiate 0% down payment with an HSP visa.
• You may start your own business. It may be in any form (sole proprietorship, company, etc).
• You may continue to reside in Japan even without a job.
• As a disadvantage, you will now be taxed on your worldwide income. More info here.

Legal Disclaimer : The contents of this article do not constitute legal advice, are not intended to be a substitute for legal advice and should not be relied upon as such. You should seek legal advice or other professional advice in relation to any particular matters you or your organisation may have.

One of Google Cloud’s less-known features is its Cloud CDN which leverages Google’s global network of data centers, servers, and ISP integration to deliver content to the end user in a quick manner. Its efficiency is further increased when coupled or used with other services in Google and Google Cloud, such as Google Domains, Google DNS, and Google Compute Instances. Since 2018, this site is hosted using Google’s Click-to-Deploy WordPress deployment (tutorial) using Google Domains (tutorial) with SSL encryption (tutorial) and maintenance via SSH to instance VMs (such as updating PHP version in Google Cloud VM Instance). Thus it is natural to leverage Google Cloud CDN, with the added benefit that it won’t break with updates to WordPress since the CDN configuration happens server-side and not WordPress-side.

Follow along with me as we explore how to activate Google CDN for your Google Cloud-hosted service.

Assumptions

This guide assumes the following:

• Your website uses Google Cloud VM instance(s) in an unmanaged instance group.
• You have access to Google Cloud Console and SSH privileges into your VM instance(s).
• Your website is HTTPS with a valid SSL Certificate.
• Access to DNS settings (whether Google Cloud DNS or otherwise).

Before We Begin

Your site should experience almost no downtime (with the exception of resolving DNS to the new load balancer global IP) if everything is configured and works correctly. This guide piggybacks off of Google Cloud’s official documentation on Setting up Cloud CDN with a managed instance group, but there are quite a few changes we need to make to align with the assumptions above, namely:

• In addition to HTTP/80, configuration for HTTPS/443 is also required.
• We are applying CDN to an UNmanaged instance group.
• Slightly different Health Check procedure.

Don’t worry, there will be reminders throughout this guide if we deviate from the official tutorial from Google Cloud.

Setup

Add Health Check Tag to Instance VM

The health check and firewall rules need to know which instances to apply the health check to, which can be accomplished easily by applying tags to the instance VM. Add allow-health-check as a tag to all instance VMs for your service:

Instance Group Settings

If not done so already, you will need to add your VM instances into an instance group. This tutorial assumes you already have a VM instance NOT created via an instance template and therefore can only be added into an UNmanaged instance group.

One extra setting you will need to do is to add named ports for HTTP/80 and HTTPS/443 to the instance group. Follow the section here, but don’t forget to also add HTTPS/443 as well.

Set Up Firewall

Follow the official guide section here to configure firewall rules for your health check. Note that our health check will use only HTTP/80 and so it is NOT necessary to add HTTPS/443. You are free to modify the name of the firewall rule but keep all other values (IP ranges and tag name) the same.

Set Up Load Balancer

Google Cloud CDN operates alongside a load balancer with forwarding rule applied to a global IP address. All three settings are initialized and configured at the same time. This is the hardest part of the tutorial, but don’t fret! I will walk you through the steps as detailed as I can.

Create a Health Check Page

The default settings for a health check as indicated in the official guide has a Request path of “/” which generally points to the main page of your website. The main page contains many content and is overkill to be used for a simple ping, so it is highly advised to create a separate page with a few words of text to minimize the overhead. We can actually use the sample code in Step 11 here to create a simple page. SSH into your instance VM and execute the following code:

vm_hostname="$(curl -H "Metadata-Flavor:Google" \
http://169.254.169.254/computeMetadata/v1/instance/name)"
echo "Page served from: $vm_hostname" | \
sudo tee /var/www/html/healthcheck.html

This will create a page called “healthcheck.html” on your root site directory containing the name of the VM instance. Note this page name as you will use it when configuring health check in just a moment.

Retrieve SSL Certificate Public and Private Keys

Because we are allowing HTTPS connection through the load balancer, at some point in the below guide you will be asked to provide the SSL certificate for your website. If you have configured your SSL certificate via Certbot (the same way I have as outlined in my other post), then SSH into your VM instance (using a separate browser tab) and execute the following commands to retrieve your public and private SSL certificate keys:

sudo certbot certificates

The output should give you something similar to the following, in which you can just view its contents to find all the certificate keys you need:

# Replace ##DOMAIN## with whatever is output in the above command
sudo cat /etc/letsencrypt/live/##DOMAIN##/fullchain.pem
sudo cat /etc/letsencrypt/live/##DOMAIN##/privkey.pem

Hold onto these outputs as you will need to copy-paste them later.

Configure the Backend Service

Follow the “Setting up the load balancer” section up until you finish Step 6 in its entirety. However, note that step 6c should be changed to HTTPS, 6d to “https“, and 6f to “443“. Refer to below figure for an example.

Check “Enable Cloud CDN” so that the system will automatically configure the Cloud CDN for you. For Health Check, since we haven’t created one yet, go ahead and follow the prompt to create new. Change the protocol to HTTP and use port 80 (remember our health check uses HTTP/80 and not HTTPS). Most importantly, the “Request path” should be configured to the health check page you created above (/healthcheck.html), relative to the root of the site directory.

Return to the backend settings and finish the rest of the configuration.

Configure the Frontend Service

Step 8 of the official guide assumes we already have an external IP address reserved, but I left it out until now because we can configure it here. You will need to configure TWO protocols – one for HTTP and HTTPS. When needing to specify the IP address, select “Create IP address” to create one on-the-spot.

For configuring the forwarding rule for HTTPS, a few additional steps are required, namely providing the SSL certificate keys. For certificate section, click on “Create a new certificate” to reach the UI below.

Here is where you take the outputs from “Retrieve SSL Certificate Public and Private Keys” section above and paste them here. Note that your “fullchain.pem” should actually contain TWO certificate sections, the first one should be copy pasted into the “Public key certificate” field and the second one into the “Certificate chain” field. “privkey.pem” only has one key which you can just copy the entire file contents and paste into “Private key”.

Below is a sample frontend configuration for HTTPS.

Follow the rest of the official guide until you finish setting up the load balancer.

Testing Load Balancer Configurations

Take several moments break for the load balancer to finish initialization. It should look like the below figure once finished. Note that the green check mark indicates the backend is configured correctly and that the health check is working as intended. If it is NOT green then revisit the above steps (and also using the official guide as reference) to diagnose the issue. For more concrete evidence, following “Testing traffic sent to your instance” section should direct you to the homepage of your website (ignore what step 5 says).

Modify DNS Settings

If the above load balancer test is successful, it is time to change the DNS settings “A” record from your previous single VM instance IP address to the load balancer global IP address. The global IP address can be found by clicking on the name of your load balancer in the load balancer settings page (Refer to Figure 3-1).

Enter your hostname into DNS Checker and verify that the global DNS servers return your load balancer’s IP address as the result. It may take a few moments for propagation. Once propagated, do a final check by accessing your site using your hostname (recommended to use incognito or private browser to test).

Congratulations! Your site is now configured for Google Cloud DNS! However, there are a few additional checks to make sure Google Cloud DNS is actually cacheing your website resources…

Test Caching Settings

Google Cloud DNS actually has a few rules on what resources can be cached, which can be found in detail here. After understanding the rules, use a caching checker such as this one, fill in your site, and look at the results. Your cached resources are listed in the “Cached Files” section, but there is a catch. Google Cloud DNS does NOT cache resources when cached via ETAG. Alternatively speaking, if you see a “Type: Cache-Control” tag with another tag specifying the expiry duration for a resources hosted on your domain, then that means your resource is properly cached using a CDN.

If you find your domain resources are not being cached, then check your WordPress Cache Settings and verify that caching settings are turned on. If the problem is still not resolved then follow the server-side guide here as well. For this site the WordPress cache settings were fine and my VM instance just did not have the mod_expires module enabled.

So here it is: a live post on tips and advice to secure and protect your crypto from falling out of your hands. Owning cryptocurrencies not only involves risk with its price, but also how it is handled – a small hack or simply a wrong transfer can cause your entire portfolio to vanish in an instant, and once the transactions are verified and carved into the crypto’s ledger one will undoubtedly be at the mercy of the recipient.

Read on to prevent the above from ever happening to you.

This guide is split into online and offline sections. The online portion mainly involves handling cryptos on exchanges but also goes into generic device security. The offline section touches more on the “human” side of how to make sure your crypto wallets won’t be hijacked by somebody else. Those who currently don’t use hardware wallets should still go through the offline section since there are specific guidances on how to secure private keys.

Online

Beware of Virus/Malware on Device

We’ll start off with one that is common knowledge to everyone but is worth mentioning anyway: making sure the device used to transfer crypto isn’t infested with viruses or malware. Assuming one uses a laptop it is important to run antivirus software frequently and clean out any suspicious software. Even keeping Windows Firewall and Windows Defender on and updated can help immensely to prevent malware from searching the file system for contents that look like private keys and transferring them across the network.

The above is a “duh!” concept for everyone who is savvy enough with computers to want to buy crypto, but many often overlook this next potential vulnerability – that browser extensions may be potential avenues for attack. Recently there are a lot of news such as this one where companies – intentionally or not – are snooping on what users type on their devices. Just because the above news is for iOS devices doesn’t mean it won’t happen on Android or laptop (which is arguably easier to develop in the form of browser extensions).

So what can one do? It is a lot of trouble to look through each installed extension and audit them for bad behavior, so one potential solution to mitigate this risk is to limit active extensions to the bare minimum via private/incognito mode on your browser to access crypto exchanges or when entering any kind of sensitive information. There is an added inconvenience of the crypto exchange not recognizing your device and having to authorize it via email verification, but it is a small price to pay for securing your assets.

Careful of Untrusted Sites and Phishing

Untrusted sites are very easy to recognize – modern browsers will blatantly tell you whether the site accessed is not secured. It goes without saying that when dealing with crypto one should never enter affiliate with any sites that use an insecure connection. There are two reasons: the first is that any viable crypto exchange or crypto website has SSL certification and is transferring data via Secure HTTP, so sites which are not are phishing at its worst and untrustworthy at best. The second reason is that insecure HTTP is vulnerable to network sniffers who can extract one’s private information – and one won’t be any the wiser.

One can check whether a site is secured usually by referring to the browser’s URL bar. The following is an example of this site secured with an SSL Certificate from the free and open-source project LetsEncrypt:

However, there is a caveat that must not be overlooked – there have been cases where fake sites acquired users’ private information by using a URL that looks very similar to the real site. By replacing a letter in the URL with a unicode character, hackers can design a front page of a crypto exchange which looks strikingly similar to the real site, and pass this fake URL via email or other methods to entice unsuspecting users. While this fake site may have an SSL Certificate registered and is transferring data via secure connection, it alone isn’t enough to identify a site as the real one.

The crypto community advises all crypto owners to bookmark crypto exchanges and other commonly used sites after verifying that it is real to prevent such a situation from happening. Additionally, never click any links inside emails if one has not requested or isn’t sure of its origin. A common example is a hacker group obtaining a list of email addresses of users on a crypto exchange and sending fake password reset emails to everybody. Remember to register for increased security such as 2-factor authentication (2FA) for every crypto exchange account so that hackers will not have access to funds even though login information is compromised.

Use Credible Exchanges

At the present circumstances crypto adoption is not high enough to the point where one can use it to exchange for goods and services. Thus, to a majority of crypto users (and barring direct exchanges with friends and family), the only way to turn crypto to something of value is still to translate it to fiat currency via an exchange. It should be a no-brainer that one should use reputable, secure, transparent, and licensed exchanges to purchase and sell coins.

Personally, my go-to options are Coinbase for exchanging fiat into crypto, and Binance for crypto-to-crypto transactions. Especially with fiat, KYC will likely be necessary in which identity, bank, and phone verifications are required. However, excluding fiat, it is likely possible to deposit, trade, and withdraw (with a limit) crypto in many exchanges such as Binance.

To summarize, check for the following very important points when picking exchanges:

• Check exchange has license to operate in the country it is based in.
• Ensure exchange has secure methods such as blocking unknown IP logins and 2FA authentication for login, trade, and withdrawals.
• Pick exchanges which has funds protection in case exchanges get hacked (believe me, it happens a LOT). For example, here is Coinbase‘s page on insurance of user funds.
• Understand the exchange’s crypto daily withdrawal limits for unverified users. For example, Binance has a 24-hour withdrawal limit of 2 BTC worth of crypto.
• Check it is legally possible to use the exchange (in case one resides in a different country or is a different nationality from the place the exchange operates out of).

Test Deposit/Withdraw/Transfer With Tiny Amount

Once a crypto transaction is signed and verified on its blockchain it CAN NOT BE REVERSED. This is an important concept one must understand when dealing with the transfer of crypto from wallet-to-wallet or wallet-to-contract. While an unverified transaction can still be cancelled, don’t bet on being able to spot the mistake and send another transaction to the blockchain to cancel it within the seconds it takes for a miner or staker to spot and write into the coin’s ledger.

To ensure the funds are transferred and received properly, it is highly advised to send a tiny tiny amount (in the range 0.01 ETH or, if using an exchange, the minimum amount they allow) and check the recipient’s wallet for confirmation of the funds. For deposit into exchanges, typically it would require a certain amount of confirmations – this depends on the exchange and the coin to be transferred – but it is fine to just only note the exchange has received and is awaiting more verifications. This safety check becomes ever so important the more value one transfers in a single transaction… don’t skip this step!

Not Your Keys, Not Your Coins

Recently more and more licensed and reputable exchanges offer insurance or customer funds protection in the event of a hack or unexpected loss of exchange funds. However, insurance will only cover for so much, and claiming the lost funds requires an extensive amount of time, resulting in a potentially high opportunity loss.

“Not Your Keys, Not Your Coins“ is a mantra widely chanted amongst the experienced within the crypto community. The individual is ultimately responsible for all of the security and storage of their crypto assets, but such responsibility can be a huge burden for some to bear. Similar to a huge amount of cash, most people are more comfortable with storing it in a bank or other authority than handling it themselves. However, crypto regulations are still lax and/or virtually non-existent in many countries around the world – laws and adoption has not yet caught up with the likes of cash. So until that day comes, one is responsible for any private keys, recovery pass-phrases, maintenance of hardware wallets, seeds, and backups of all crypto wallets in possession.

Crypto can be the Next Big Thing™ to happen and may make one rich, but such doesn’t come with a proportionate amount of risk and responsibility. Many articles and guides (and including the one you’re reading right now) attempts to help newcomers make their first dip into the crypto universe, so it is important to understand what is at stake and to properly guard against all possible risks.

Keep Long-Term Holdings in Cold Wallet

A cold wallet (or cold storage), otherwise known as hardware wallets or offline wallets, have the added requirement that they be connected to a laptop or other device in order to access the funds inside it. It is an extra layer of security due to two reasons: one must have the physical wallet device to interact with their crypto funds, and the private keys for crypto wallets inside are protected and can never be retrieved (instead, if the cold wallet is lost, a seed is required to restore its contents). Just like how one wouldn’t keep thousands of dollars in their wallet when going out, one should keep long-term HODLs in a cold wallet and lesser funds in a hot wallet/exchange to react to price changes.

A good practice is to assume that contents in any hot wallet (exchange funds also count as being stored in hot wallets) may be lost in its entirety due to some unexpected event, which may include but not limited to any of the following:

• Leakage of private key
• Hacking
• Theft
• Forgotten password
• Lost cold wallet device
• Leakage of cold wallet passphrase/seed

DeFi and Staking

Lastly, a note on the risks of DeFi and staking. For those who may not know, DeFi is short for “Decentralized Finance” and is an adaptation of financial instruments for the crypto universe. Things like P2P loaning, lending with collateral, and others are possible in a transparent (via code) and decentralized manner. The ability for some cryptocurrencies such as Ethereum to maintain contracts allows for such projects to exist, and is an alternative way to generate dividends and interest on your crypto investment.

Staking, which is more commonly known to everyone, is simply securing the blockchain by offering ones coins as collateral when validating transactions. Penalty in the form of losing rewards and sometimes losing a portion of the staked coins only happen when the blockchain identifies one being a bad guy and attempting to overwrite the contents of the ledger. Staking ones coins is normally safer than dealing with DeFi because staking is a basic feature of a coin which supports Proof-of-Stake (PoS) whereas DeFi involves the additional development and usage of smart contracts.

That said, in addition to the risks associated with staking and using these financial instruments in a traditional, non-crypto sense, there are some extra points to consider when delving into the crypto world of passive income. For the realm of staking, it is best to search for coins in which the funds never leave one’s wallet, such as Tezos, because of the “Not Your Keys, Not Your Crypto” principle mentioned in an earlier section. Other coins, such as the newly Stake-able Matic from end of June 2020, require that funds be transferred to a delegate, with smart contract commands to re-invest earnings or withdraw funds or un-bond from the delegator. In other words, funds leave your wallet in these types of staking, so be careful of any potential exit scams especially for not-so-popular coins (BitConnect comes to mind – although not a staking scam, BitConnect did require one to lock up funds for a minimum of 120 days). Use similar judgement when eyeing DeFi projects.

Offline

Secure Your Private Keys and Seeds

The ultimate key to securing one’s crypto funds is to ensure nobody has access to one’s private keys or recovery seeds/passphrases. Often the way to do so is to encrypt them using a tested and standardized encryption mechanism. There are many cryptographic libraries available to use, but a simple yet effective way is to use RAR. To encrypt file(s), use the following command in the terminal:

rar a -hp [NAME OF RAR FILE] [PRIVATE KEY 1] [PRIVATE KEY 2] ...

This ensures that both the filenames and all contents are encrypted. As an added bonus, set the name of the RAR file to something not related to crypto private keys to reduce suspicion.

To decrypt a rar file, use the following:

unrar x [NAME OF RAR FILE]

Decentralize Your Secured Keys

Now that a secure, encrypted file containing private keys and seeds have been created as mentioned above, it is time to consider the various secure places to backup. To reduce the risk of a single point of failure, decentralization of your keys is highly important. The following places can be ideal to place a single backup to – the more places the better:

• Locally on laptop
• To an external HDD
• Google Drive (BONUS: enable 2FA or phone authentication for Google account)
• To a secured USB drive, to be carried around with your house/car/etc keys.

Have a Backup, Unused Hardware Wallet

If the hardware wallet happens to be misplaced and funds need to be accessed immediately, then a way to restore your crypto wallets within a hardware wallet is to input seed/passphrase generated from the previous wallet into a new hardware wallet. Having to purchase another one requires a lot of time, so be sure to have a spare, unused hardware wallet for such situations.

NEVER SHARE YOUR KEYS WITH ANYONE

Just don’t. Never trust anyone else with your private keys. Enough said.

This is the beginning guide to a trilogy of Taiwanese Mahjong, going into statistics and numbers before delving into player behavior and playing scenarios. I don’t think other places in the internet has quite as detailed analyses (in English), and I will continue to add to it for, as I have always mentioned in other of my posts here, personal learning and memorandum.

Differences With Other Forms of Mahjong

• 16-tile hand instead of 13. Players need an additional set and goes out with 17 tiles.
• A player can win with a “chicken scratch” hand (0 tai).
• A “moving” dead wall in which 16 tiles are maintained at all times (Replacement tiles are taken from this dead wall, in which an equal amount should be added from the end of the live wall). Kongs add an additional 2 tiles to this dead wall.
• The dealer, starting out, gains or loses 1 tai if they win or lose, respectively. On wins or cases where nobody wins, the dealer retains their dealer status and wins/loses 2 additional tai per streak count. For instance, a dealer who won twice but loses afterwards will need to pay 1 + (2*2) = 5 tai to the winner.
• When a player goes out on a self-draw, every loser pays the winner equal to the amount of tai of the winner’s hand. If the dealer is not east, that dealer will need to pay additional tai equal to dealership (1 tai) plus any consecutive streaks they have.
• In cases where a discard results in multiple players going out, the closest person (counter-clockwise from the person who discarded the winning tile) is awarded with the win.
• Concealed Kongs – a player may choose to conceal their kong if they have all four tiles in their hand.
• One cannot meld a Kong from their left player – Pung should be declared and the last tile added to form a Kong on the player’s next draw.
• One may choose not to pick the last tile. The match will still end with nobody winning, and dealer retains their seat and +1 consecutive “streak”.

Probabilities and Statistics

Average number of cycles per round

There are 144 tiles total (136 + 8 flowers) in Taiwanese Mahjong. 16 tiles are reserved for the dead wall at all times. Each player also begins the game with 16 tiles (the dealer begins by drawing the first tile so we consider the dealer having 17 tiles starting out). On average, 4-5 flower tiles are revealed before the dealer makes their first discard.

With all of the above added together (16 dead wall + 65 combined tiles in 4 hands + 4-5 flower tiles revealed), only 58-60 tiles remain to be picked. 4 tiles are picked each cycle which means that there are effectively 14-15 cycles in a single round. This is important when going into detail about strategies in early/mid/end game.

Average rounds per game (4 winds)

A single game (一將/一雀) normally consists of 4 winds, which means that there are a total of 16 dealers (4 per person). Assuming that every player is equivalent in skill, the odds of going out is assumed to be 25%, which means that the amount of times +1 consecutive dealership is achieved is 16 * 0.25 = 4 times. Within those four times there’s another 25% chance that the dealer wins yet again, which brings our average total rounds per game to 21, but is widely known to be 20-22. This means that, statistically, a single player, on average, wins about 5-6 hands per game, with 1-2 hands within that being a self-draw win (everyone pays the winner). Unfortunately, this also means that a single player will be on the losing side of a self-draw 4-6 times.

Why is this important?? Because if a player wins 4 games (with around 1 of which being self-draw to earn more points) to cover up for the times of losing to a self-draw, that player will not lose the game and will finish positive. However, this is valid only if you do not lose by discard. The point is, if ones goal is to finish positive, one can attack to win 4 times (provided their starting hand is quite favorable) while playing more defensively for the remainder of the 16-18 rounds.

There is a method for finding out which starting hands are favorable and which are not, which will be left for a later guide.

Picking tiles

With 3 suits, 4 winds, and 3 honor types in mahjong, the chance of picking a desired tile stands at 1/34 (2.9%). Additionally, although there is not enough evidence to back this claim, it is believed that you will, on average, pick a tile you need to complete a set every two picks. Because there are 4 of the same tile, the probability decreases by -0.75% for each desired tile seen (to 0% if all 4 tiles are accounted for). There’s no need to remember this particular probability, but it does somewhat play into the next point:

Starting hand composition

Taiwanese Mahjong is played using 144 tiles with a 16-tile hand. On average, when a round begins one should expect to see 4.2 tiles of a single suit and 3.4 tiles of wind/honors. This means that an ordinary hand will have roughly 4-5 man tiles, 4-5 bam tiles, 4-5 circle tiles, and 3-4 wind/honor tiles. Likewise, there is a 9/34 (26.47%) chance to pick one of bamboo, circle, or man tiles, and 7/34 (20.58%) chance to pick a word tile.

This is an interesting and important concept to understand when going into detail about play strategy; for instance, a player with many non-pair winds/honor tiles will likely start the first few cycles of the round discarding them, whereas those who start discarding suit tiles very early may have a higher amount of “gates” and be closer to going out.

[Early Game] Discarding Word tiles vs 1, 9 tiles

This part may be controversial and not everybody may agree with the reasoning, but in the discarding phase it is better to discard word tiles over an “orphan” or . This is because to create a set using , for instance, any of the potential three remaining , four , and four may be drawn from the live wall. Compared to an orphaned word tile such as (only three remaining), the probability of making a meld-able set (any of the following , , or ) is three times higher than picking a second word tile .

This is a controversial concept because some players may take the chance to pick into a paired word tile provided one hadn’t already been discarded, and that the chance another player has a pair of the same word tile you have is minute, and thus if such opponents aim to go out then they will have no choice but to discard it somewhere down the line. That said, however, the following negative possibilities exist which diminishes the chance of one picking and melding word tiles in mid/late game:

• Opponent(s) want to discard the tile but found out they shouldn’t due to fear of discarding the winning tile.
• One or more of the tile resides within the dead wall.
• Another opponent also has a pair of the same word tile.

The above does not include the simple fact that any player can go out with a 0-tai “chicken scratch” hand, so the aim for nearly all ordinary hands is to reach tenpai status as early as possible while waiting for a suitable set of tiles.

Thus, the most optimal discard order for the early game is as follows:

1. Wind tiles that are not the prevalent wind nor your position’s wind. This is to account for the chance you may pick a wind or honor tile while still in the early game which has potential to earn you an additional tai.
2. Remainder of orphan wind/honor tiles.
3. Orphan ‘s and ‘s.
4. Other separated tiles depending on situation.
   

Characteristics of Suit (Number) Tiles

The Principle of Symmetry (對稱性)

The first thing to understand about number tiles is that the set of 1-9 tiles is symmetric. This means that 1 to 5 is the same as 5 – 9 but “flipped” – every characteristic of 1 can be applied to 9, every characteristic of 2 can be applied to 8, and so on. For instance, discussions revolving probabilities of can be exactly applied to .

This principle is very convenient when discussing probabilities and tile logic because one need only look at half (i.e. tile numbers 1 through 5) of the number tiles. From henceforth, barring some side notes, I will use tile numbers 1 through 5 for discussion around number tile logic and probabilities.

The Principle of Suppression (向下壓/向上壓)

The principle of suppression states that, especially in the beginning cycles, a player who discards a numbered tile will likely not want any numbers smaller than it (bigger if talking about 5 through 9). For example, a player who discards a is likely to not want a , and likewise discarding a shows they likely don’t want or . This is true because players want to discard all tiles not suitable for melds in the early game – starting with those numbered tiles furthest away. Of courses, there are cases such as in which a player may discard but normally one chooses to keep it early game in case the left player discards a which can be used complete a meld. One can learn a lot about their opponents’ hands when they discard tiles using this principle.

The “1-4-7” Rule (一路性)

1-4-7:

2-5-8:

3-6-9:

Because mahjong incorporates the Chow mechanic, there is an observation which states that discarding a or is somewhat safe to an opponent who discards a because by discarding the opponent shows they do not have the tiles necessary to meld or to complete a Chow. The same rule can be applied to 2-5-8 and 3-6-9. However, this does not mean that when an opponent discards a that they won’t want a . It is possible the opponent can chow or win with this tile, but the probability that they want this tile is roughly halved. Obviously, throwing a does not have any significant impact on the chances the opponent wants .

The 1-4-7 rule has another interesting characteristic in that it can be leveraged to perform a 3-gated wait. Assuming one already has 3 sets and a pair of eyes (not shown):

One can wait for , , or to win, increasing the amount of wait-able tiles to 11 instead of 8 for a 2-sided wait. Sometimes for higher than 2-gated waits I may choose to not win on a first discard but instead go for a self-draw victory due to the increased chance of success.

D-A-D (Develop – Attack – Defend)

After noting the above sections, we can know go into the philosophy many professionals refer to as the D-A-D strategy. While many online sources go over each section and how it is beneficial to follow the strategy, I haven’t found any which goes into the fine details and reasoning behind why it is so. I will do my best to explain the logics behind this strategy here.

Let’s dive straight into the first “D”, which is:

Cycle 1-6 : Develop (序盤)

The first six cycles is referred to as “D” which stands for Develop, otherwise known in other sources as “Opening” or “Early Game”. Earlier in the “Starting hand composition” section we know that on average a player’s starting hand consists of 4-5 tiles of each suit and 3-4 word tiles. Accounting for the probability of picking an additional 1-2 word (remember there is a roughly 20% chance – 1 in 5 picks – in which a word tile is drawn) tiles plus discarding un-needed orphan tiles, cleaning out your hand to a point where one can begin to attack averages at around 6 rounds. In other words, if one still has orphaned cards after the Develop phase then they may consider giving up for this round, skip the attack phase and go straight into defense.

Cycle 6-12 : Attack (中盤)

The Attack phase (mid-game) can be further split into two:

Cycle 6-9

Players with a good starting hand or has had favorable melds will reach tenpai within the early mid-game. Other opponents who see that they have many melds should watch this player’s discards carefully as it can give clues on what they are waiting on. If one is not tenpai and does not have a safe tile to discard then they can consider folding and playing defense for the rest of the round.

Cycle 9-12

Most players with an ordinary starting hand and luck will likely reach tenpai in this stage. At this point one should weigh the risks of going for the win vs how strong opponents’ hands are. Waiting on a single tile or a blocked 2-gated wait is not ideal at this stage.

Cycle 13-15 : Defend (末盤)

If one is not tenpai at this end-game then give up, because no matter what it is not appetizing to discard a dangerous tile when one is not close to winning. Give that chance to others who are perhaps not as experienced at you are, or believe that the opponent(s) will not win on self-draw due to being stuck or merely waiting on bad tiles.

Good vs Bad Starting Hand

Like with other types of mahjong, the most convenient method is to check how many cards you are away from tenpai. More experienced players are able to adjust the formula to further calculate how easy/hard it is to acquire the remaining tiles necessary – some pointers and advice is left for a future guide on advanced gameplay. For now we will just go over the simple formula, which is how many tiles you are away from tenpai.

• 2-3 to tenpai (Good)
  Assuming from above that on average it takes two draws to bring one’s hand a tile closer to tenpai. Thus, 2-3 away from tenpai means one will reach tenpai before mid-game, in which case you should forcibly attack and play to win.
• 3-4 to tenpai (Normal)
  With this hand you should expect to have either 1 set and 1 pair, or 0 sets and 1 pair but with easy to meld combinations. Expect for everyone to reach tenpai 8-10 cycles into the round. Obviously, because this type of hand is the most often to occur, one should consider the situation and decide whether to continue attacking or to switch to defense during mid-game.
• 5+ to tenpai (Bad)
  One should seriously consider defending for this round for any hand above five tiles needed to tenpai. Unless the player has amazing luck drawing tiles with their left player feeding them, one would already be at the last three cycles before reaching tenpai, which also means one would need to take 2-3 rounds discarding risky tiles mid-game to get to that point. Withdrawing like this for even a few rounds out of 20-22 rounds is not going to hurt the overall game. There is a saying amongst professional players which goes along the lines of: “Anyone who discards the winning tile (lose) during the end-game (末盤) without tenpai does not know how to play mahjong“.
  

Disclaimer (2020-06-03): The first half of the guide (first page of registration) has now been updated to the latest version. The second section about Credit Card information has been left as-is since its UI has been left nearly the same since two years ago. Ignore the sudden jump in numbering as I have only updated the first half of the guide. That said, if there are any discrepancies please do let me know in the comments section!

Introduction

Rakuten Bank is an online service that aims to provide various banking services to customers. Those who use or subscribe to various Rakuten services will often enjoy additional benefits for their banking needs. Just like with Rakuten Card, there are no physical offices you can go to so everything is managed online instead. Having an account with Rakuten is recommended if you heavily use things such as Card, Securities, Edy, or Super Points. In any case, you can read about a non-exhaustive list of advantages and disadvantages below:

Advantages

• All services available over the internet
• Enjoy additional benefits when linking bank to many other Rakuten services.
• Deposits can be done at Yuucho, and withdrawals from Yuucho and many popular convenience stores.
• It’s possible to have up to 5 transfers and 5 withdrawals free per month based on your “Happy Program” ranking.

Disadvantages

• If you ran out of free transfers and withdrawals, the fee can be pretty high (especially weekends/holidays)
• Extra work may be necessary to link other company’s services to Rakuten Bank (for billing/transfer purposes)
• Interest returns on your balance are lower compared to other companies.

Prerequisites

You, obviously, need to reside in Japan and also have at least one of:

• Japanese Driver’s License
• Health Insurance Card
• Residence Card

After you submit the application you will be asked to submit proof of document for one of the above. This can be done via mobile application or via snail mail. For obvious reason I recommend using the Rakuten Bank mobile app to upload images of your formal document of choice.

In addition, if you haven’t registered for a Rakuten group account yet, then I strongly advise you do so as the below guide is tailored for those who already have an account. Don’t worry, the registration should be very straightforward (Last/First Name, username, password, e-mail address).

Before You Start…

Don’t stay too long on a screen or else it will TIMEOUT and you will have to do it All. Over. Again. If you are slow in filling out Japanese forms then I recommend you look over this guide first and prepare in however way you wish before proceeding.

Application Process

Getting to the Application Form

Click here to support me (thank you!) and start the process of signing up for Rakuten Bank. Click on the red button in the middle of the page.

If you have not logged into your Rakuten account yet, do so. If you do not yet own an account, go ahead and create one by clicking on the right-side button.

You will be transitioned to a temporary page stating you have successfully entered into the campaign, and will be redirected to the signup form in about five seconds.

Form Part A: Bank Account Info

Refer to the notes below the images for additional information:

名前：漢字) Your name should be input, in FULL-WIDTH ALPHABET, your name as shown on your residence card. If you’re like me who has strange characters in your name (such as a hyphen in first name), then omit it completely.
(eg “SMITH, ANNA-MARIE” should be “SMITH, ANNAMARIE”)
名前：フリガナ) If you registered your Rakuten group account using English name then this should be autofilled with “ー　ー”. You can choose to leave as is or replace with Katakana reading.
性別) Left for Male, right for female. Nothing confusing here.
生年月日) Birthday. Nothing special here.
電話番号) Enter your phone number.
メールアドレス) This should be autofilled for you if you logged into your Rakuten group account. Just re-enter e-mail in confirmation box.
住所：郵便番号) Input your residence’s zip code and click button to the right.
住所：都道府県市区町村) Enter the rest of your address as printed on the back of your residence card.

Note the checkbox, which translates to”Please check this box if you do not have Japanese citizenship. Those with Japanese citizenship need not check this box.”
Hint: you should check this box.
「米国人」…) Check right box if you are American. Otherwise, check left one.
居住地国の確認) Check the above box if you only reside in Japan. In other words, unless you have tax responsibilities from staying in other countries, check the top box.
職業) Below is a formatted table of the selections:

(Depending on what you chose in #7 above, you may have to fill out the below. Otherwise just move on.)

Basically follow the notes directly on the image and fill in your company’s information as best you can.

Pick the kind of card you want. The types of cards go from debit cards to credit cards to simple bank (cash) cards. Keep in mind that you may need to provide additional information if you choose to get a cash card with credit option. Refer to the other link for how to register for Rakuten Card here for some additional references.

講座の利用目的) Must check one of the options above. Some common ones are:
生活費の払い: Personal expenses
給与/年金の受取: Salary Transfer
楽天サービスの利用: Use Rakuten Services

Select the type of document you will provide as personal verification, in the form of the following:

楽天銀行toto) Rakuten Toto is the lottery business of Rakuten. It is not a required service so uncheck it if you do not need it.
楽天証券講座) Left box if you want to also create a stock brokerage account with Rakuten, otherwise check right box.

Check this required box to agree to the T&C. Afterwards click the “次へ” button on the bottom right.

Form Part B: Credit Card Info

Refer to the notes below the images for additional information:
Fill out #25-#27 as noted in image above.
28) Name in ROMAJI (Again)
29) Phone number
30) Maybe can leave blank

34) Here’s another table, in the same ordering:

35) Put yourself in the count (so if you’re single with no dependents, put “一人世帯”)
36) Tables are fun!

42) From left to right: working, student, unemployed, earning retirement benefits
43) If you earn your own income, check left. If you don’t, choose right.
44) Your yearly income (before tax!)
45 – 50) Same as the company information you may have filled out earlier on page 1.
51) Job Role:

52) Type of Employee:

The text below selection reads: If you are a student doing part-time work, fill in “Part-time”.
53) Industry:

Confirm the following two points and click red button on bottom of page.

You have now reached the confirmation page (you are almost done!!)

Form Part C: Confirm + Submit

Make sure to check the box as shown (you can leave the others as-is – they’re not subscriptions to anything or whatever that will spam you)

Aaaand, you are done! You should receive an e-mail with the confirmation receipt number. Now it’s just a matter of waiting!

What To Do After

You are required to submit proof that you reside and work in Japan. You have the choice between multiple documents you can submit. The two most commonly used documents are your Residence Card or your Health Insurance Card. Choose one you prefer, download the Rakuten Bank App, and submit front/back images of your document through it. (Unfortunately I don’t have a test account so I cannot create a tutorial for this, but there’s no rush so please ask a trusted Japanese friend to help you!)

Search “Rakuten Bank” or “楽天銀行” on your phone, or scan the QR code on the receipt screen (OR, you can even scan the QR code in the above image!)

Trivia/Nice-To-Knows

Will add as I think of them.

Visiting Taiwan and want to drive but forgot to apply for an International Driver’s Permit? If you have a Japanese driver’s license then it is possible for you to drive in Taiwan without having to transfer your license to Taiwan D/L or needing an International Driver’s Permit (IDP). On top of that, unlike IDPs which requires one to process before embarkation, those who are too late (already in Taiwan) are still able to process the necessary documents to drive – and it will be available to use immediately.

Read on for details, procedures, precautions, and other important information regarding driving in Taiwan with a Japanese D/L.

Before You Begin

Since September 21, 1996 Taiwan and Japan have entered into a reciprocal agreement (TW) which enables license holders from one country to drive in the other withOUT transferring of license. While there are some restrictions regarding vehicle types you can drive depending on the license type (I will go more into detail about these restrictions below), it is pretty reasonable and you should expect to be able to drive the same kind of vehicle as you would in Japan.

However, this article only applies to you if you do not plan to apply for an IDP or if you are visiting Taiwan under a Visitor’s Visa – this is because it is better to just translate your Japanese D/L to Taiwan D/L if you are a resident of Taiwan or otherwise have a work permit (ARC).

In addition, note that the period of validity is limited as follows:

• Usage period is 365 days from date of arrival to Taiwan. This 365 days is refreshed when you depart and revisit Taiwan.
• Your driving eligibility is revoked after the expiry date on your Japan D/L.
• If your Japanese D/L is expired, you must apply for a new translation after renewal.

Remember to bring your Japanese Driver’s License with you to Taiwan. Additional required documents are detailed in the next section.

Required Documents

Police officers and other officials who need to check your identity and license verification would need the following items. Note that you MUST have all three documents with you in order to operate a vehicle:

• Passport
• Japanese Driver’s License
• Translation of Japanese Driver’s License to Traditional Chinese
  

This translation of your Japanese Driver’s License is likely the only item you don’t have on hand, but rest assured it is possible to produce this document whether you are in still in Japan or already in Taiwan. It is highly recommended you secure this document while in Japan since there are very few places where you will be able to translate your Japanese Driver’s License in Taiwan. This brings us to the next topic…

Where to Translate Your Japanese D/L

The Japan Automobile Federation (JAF) is the official and only entity which can provide a legal translation of your Japanese D/L to Taiwanese D/L.

In Japan

JAF has many locations across Japan where you can either walk-in or mail your application to. Refer to the “日本運転免許証の中国語翻訳文（台湾）の発行が可能な支部” section to find the office most convenient for you. Do keep in mind that some offices do not accept mail-in services and others may not have a reception desk – make sure to check the “受付” and “発行” sections for each office before visiting or submitting your application. The translation will take some time, so don’t expect to be able to receive your translated document on the same day.

Bring/submit the following when applying for the translation document:

• Translation Application
• Japanese Driver’s License
• 3600JPY

More detailed information can be found on their official page. Allow yourself enough time to submit and receive the translation before embarking on your trip to Taiwan.

In Taiwan

There are only two JAF offices in Taiwan (Taipei and Kaohsiung) so you will need to make a trip to either of those cities if you wish to translate your Japanese D/L. Be careful that you don’t visit during weekends or holidays as they only operate during business days.

There is an additional document when processing in Taiwan, so do exercise caution and make sure you prepare all of the following documents:

• Translation Application – You can fill this out at their office
• Japanese Driver’s License
• Passport
• 590 NTD (~2145JPY, actually cheaper than doing it in Japan!!!)

Restrictions and Precautions

The types of licenses between Japan and Taiwan differ and do not exactly match. As such, it is important to note your translated license and refer to this document (JP) to understand exactly which types of vehicles you are allowed to operate in Taiwan.

As an example, my type of license (普通) grants me the ability many 4-wheel vehicles. A highly generic summary is outlined below:

• Generally all vehicles less than 3500kg, though there are a few exceptions
• Electric bikes
• Motorcycle class under 50cc power

Especially take note that you MAY NOT OPERATE MOTORCYCLES OVER 50cc. Most rental motorcycles from Gogoro and Wemo offer bikes over 50cc so don’t expect to be able to rent one. In addition, motorcycles have quite a different set of road rules compared to 4-wheel vehicles you must follow. In my honest opinion this restriction is perfectly reasonable especially if you have only driven 4-wheel vehicles and have no experience with bikes. Don’t drive bikes in Taiwan when you have the option to drive a compact car instead.

In Case of Accident

Driving in Taiwan is not as orderly as in Japan – you will often find that drivers’ attitudes are vastly different, bikes will weave around you, number of lanes will change immediately after intersections, signal lights work somewhat differently, and many other unique nuances:

• Contact the police (110)
• If injured, contact the ambulance (119)
• If vehicle was rented, contact rental company
• If you are fined, contact Japan-Taiwan Exchange association

For the last point above, for the area of Taiwan above Taichung/Nantou and Yilan, contact the Taipei branch:

Phone (Business Hours): 02-2713-8000
Phone (Non-Business): 0937-043-408
Homepage: https://www.koryu.or.jp/about/taipei/

For other areas of Taiwan, including Penghu, contact the Kaohsiung branch:

Phone (Business Hours): 07-771-4008
Phone (Non-Business): 0929-228-458
Homepage: https://www.koryu.or.jp/about/kaohsiung/

Automation is my hobby. Admittedly I never thought that I would be interested in it when I studied computer science and engineering back in my university days (my father said my personality befits a software engineer, and I blindly followed his advice). However, during my journey to become a competent software developer I invested much time in simple scripts to do repetitive work for me. While most of the above examples involve automating games (via something like Sikuli which can control your screen), lately I have been turning my attention to scraping websites and integrating APIs for data aggregation. This post contains some of the tips and tricks I have used in order to make your web crawler quicker and more robust.

Before We Start – Requests vs Webdrivers

Many who write programs to crawl webpages often use Webdrivers such as Selenium to retrieve HTML contents. However, it can be a bit of an overkill if the pages you want to crawl are more static. For webpages with little to no dynamic content, using simple CURL libraries (such as Requests for python) for GET/POSTS will be sufficient for your needs.

Webdrivers are recommended for times when it is infeasible to purely use CURL for driving dynamic sites and content. This is especially true for users who aren’t scraping for information but merely wish to automate a series of actions on a website.

The remaining sections below contain advice for both CURL-based and Webdriver-based solutions. Parts that can be better explained with examples will contain sample code for reference – though a word of caution: Browsers and Webdriver API specs change frequently so they are not guaranteed to work in the future.

Tips and Advices

Python Libraries for Parsing HTML Documents

My goto libraries for processing HTML documents are a combination of html5lib, lxml, and xml.etree.ElementTree.

The following code’s “buildHtmlTree()” retrieves the HTML document from a URL and converts it into a tree ready for element traversal:

import xml.etree.ElementTree
from lxml import etree
from lxml import html
import html5lib
import requests

class RequestTooManyTimesException(Exception):
    pass

class Html5Scraper(object):
    def __init__(self):
        self.current_url = ""
        self.session = requests.Session()

    def buildHtmlTree(self, url, encoding='UTF-8'):
        self.current_url = url
        html_elements = self.parseHtmlContent(self.getResponseText(url, encoding))
        html_tree = etree.ElementTree(html_elements)
        return html_elements

    def parseHtmlContent(self, content_string):
        # html5lib supports tags that are not closed, so parse twice to
        # convert from html5lib's xml.etree.ElementTree to lxml's
        # lxml.html.HtmlElement.
        html5_etree = html5lib.parse(content_string)
        html5_string = xml.etree.ElementTree.tostring(html5_etree, encoding='UTF-8')
        return html.fromstring(html5_string, parser=html.HTMLParser(encoding='UTF-8'))

    def getResponseText(self, url, encoding='UTF-8'):
        response = None
        retries = 0
        while not response:
            try:
                response = self.session.get(url)
            except requests.exceptions.SSLError:
                retries += 1
                if retries > 10:
                    raise RequestTooManyTimesException("Max connection exceeded when accessing: %s" % url)
                time.sleep(60)
        response.encoding = encoding
        return response.text

Once you have the HTML document parsed into a tree, finding elements on the page becomes simple as follows, using xpath:

scraper = Html5Scraper()
tree = scraper.buildHtmlTree('https://www.google.com')
element = tree.xpath("//div[@id='mngb']")

[WebDrivers] Run in Headless or Disable Images

PhantomJSDriver is a selenium module which will emulate a headless (ie no Window) browser. It is particular useful in cases where only a terminal can be used (and no screen emulator software is installed). Because it is headless there is no overhead with regards to graphics and loading images, so unless you are doing UI testing and need to take screenshots of what webpages look like, it is not likely that you absolutely need to have a visual browser.

However, that said, PhantomJSDriver is not without its faults. This particular webdriver is not of the IE/FireFox/Chrome type, and therefore there may be compatibility issues when visiting some sites. When PhantomJSDriver couldn’t parse a website correctly or otherwise just doesn’t do the trick, more common browsers such as Chrome (via ChromeDriver) may handle that site well. Chrome is non-headless, but you may still turn a few things off – image loading included – to increase its performance. Below is a sample Python code of how to set up Chromedriver without image loading, plus a few other settings turned off:

from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
preferences = {"profile.managed_default_content_settings.images": 2}
chrome_options.add_experimental_option("prefs", preferences)
chrome_options.add_argument("--ignore-certificate-errors");
chrome_options.add_argument("--disable-logging");
chrome_options.add_argument("--no-proxy-server");
chrome_options.add_argument("--disable-client-side-phishing-detection");
chrome_options.add_argument("--disable-sync");
chrome_options.add_argument("--disable-component-update");
chrome_options.add_argument("--disable-default-apps");
chrome_options.add_argument("--disable-infobars");
chrome_options.add_argument("--disable-web-security");
chrome_options.add_argument("--safebrowsing-disable-auto-update");
driver = webdriver.Chrome(chrome_options=chrome_options)

Refer to this page for information about each of the above arguments.

Search For Elements in ID -> Name -> Class -> CSS/XPath Order

A well-structured website should have most of its important elements tagged with a unique ID – thus it is most optimal to scan an element using its ID, followed by its name, followed by class, and lastly its other attributes. The further away from ID you search for an element the more brittle your program gets, and can lead to much time debugging whether the element you picked is what you are actually looking for. In addition, even a small change in the website UI can cause your program to fail to find the correct element.

Assuming “driver” is the object holding your webdriver, elements may be searched using the following sample methods:

driver.find_element_by_id("id")
driver.find_element_by_name("user")
driver.find_element_by_class_name("class")
driver.find_element_by_css_selector("#root > div > div.css")
driver.find_element_by_xpath("//div[@style='color: red']")

Refer to this page for information about locating elements. There are other methods to search for elements and I encourage you to learn and figure out which method is best for you. Also, there are separate methods for selecting multiple elements fitting a certain criteria – this is useful for parsing information from tabular structures such as tables and lists.

Don’t Click if You can “GET”

Webdriver users may get into the habit of clicking a link to move between pages. The advantage of clicking is that one does not have to worry about how to build the GET or POST request, which can save a lot of time when wanting to create a quick-and-easy solution. However, clicking an element increases risk and overhead because changes to the element may cause the program to fail to find it, and there may be unnecessary Javascript or other tasks the browser may run upon clicking a link.

If it is the simple matter of moving to another page and not having to worry about anything else, then GET-ing the URL is quicker and less prone to error. For POST, an inspection of the network when submitting the form can give you an idea of what to include in the form parameters, as illustrated in the below figure:

requests.session() to Store Cookies

Python’s requests API, in its simplest use case, can be thought of as a wrapper for CURL. Each call of requests.get() simply retrieves the source of the HTML document but ignores any session variables and cookies in the response. Fortunately requests also allows for sessions which can be used for exactly this purpose! The following is a simpler code of what was provided in the “Python Libraries for Parsing HTML Documents” to illustrate the simplicity of how sessions work – this is especially useful when parsing sites which require users to login:

import requests

session = requests.Session()
r = session.get('https://www.google.com')