Merging git repositories - preserving history

Repository A:

File1
File2
File3

Repository B

Filea
DirB

We want to import Repository A into B as a sub directory

First move all the files in Repository A into a subdirectory:

Repository A:

mkdir imported
git add imported
git mv File1 imported/
git mv File2 imported/
git mv File3 imported/
gt commit -m"moved fiels into subdir"
git push

Repository B

git add origin repb
git fetch
git merge --allow-unrelated-histories repb
git push

Hopefully now RepA appears in RepB in directory imported  !

Installing Visual Studio Code on linux - non-root user

Based on a stack overflow answer:

1. Download Visual Studio Code for Linux
2. Copy it somewhere you are happy to extract it into and keep eg ~/bin
3. Extract it: tar -xvf VSCode-linux-x64.tar.gz
4. Add the executable to the system path eg export PATH=$HOME/bin/VSCode-linux-x64
5. Run the code executable to open Visual Studio Code
6. (Optional) add the export statement into your ~/.bashrc


Installing node.js on a linux machine as non-root user

Follow these instructions to install node on a linux machine; no root privileges required.


We will install nvm (Node Version Manager) to allow us to install several versions of node onto the same machine:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash

This currently produces output as the following:

Resolving deltas: 100% (4737/4737), done.
* (HEAD detached at v0.33.1)
  master
=> Compressing and cleaning up git repository
Counting objects: 7495, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (7440/7440), done.
Writing objects: 100% (7495/7495), done.
Total 7495 (delta 5009), reused 2259 (delta 0)
=> Appending nvm source string to /homes/sms67/.bashrc
=> bash_completion source string already in /homes/sms67/.bashrc
=> Close and reopen your terminal to start using nvm or run the following to use it now:
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm

Now add the following to your ~/.bashrc file (last two lines of the output above):

echo "export NVM_DIR=\"\$HOME/.nvm\"" >> ~/.bashrc
echo "[ -s \"\$NVM_DIR/nvm.sh\" ] && \. \"\$NVM_DIR/nvm.sh\"  # This loads nvm" >> ~/.bashrc

Then source your .bashrc

myhost: source ~/.bashrc

The help page for nvm can now be found by:

myhost: nvm --version

The command above hopefully returned soem output (how to page).  If so great we can now install a version of node:

myhost: nvm install node

Then to use our installed version:

myhost: nvm use node

Check your installed versions of node and npm using:

myhost: node -v
myhost: npm --version 

You should now be able to create your first node application, or write some protractor tests!

Git - configuring files with execute permissions

To configure files so that they are executable when cloned/checked out from git, (saving you to chmod after a checkout):

git add

git update-index --chmod=+x

git commit 

There may also be a method you can use via a .gitattributes file?!

Docker compose 'tricks'

Access a compose container in interactive mode

see: solution on docker forum

Having started docker-compose using a compose file similar to:

version: "2"
services:
  selenium:
      restart: always
      image: stesho/selenium-remote
      ports:
        - "4444:4444"
  centaurtestservers:
      restart: "no"
      image: stesho/centaur-testservers
      ports:
        - "4004:4004"
        - "4005:4005"
        - "4006:4006"
        - "4007:4007"
        - "4008:4008"
      volumes:
        - B:\docker-mount\centaur-build:/tmp/centaur-build
      command: bash -c "echo 'hello world'"
  centaurtests:
      depends_on:
        - "selenium"
        - "centaurtestservers"
      restart: "no"
      image: stesho/centaur-tests

I can run docker ps to list the container now running:


P:\>docker ps

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS                          PORTS                                             NAMES
5b32cca41c49        stesho/selenium-remote       "echo 'hello world'"     2 minutes ago       Restarting (0) 38 seconds ago                                                     centaurtests_selenium_run_1
c046fd91bc9f        stesho/centaur-testservers   "/bin/sh -c '/bin/..."   4 weeks ago         Up 4 weeks                      3004-3008/tcp, 0.0.0.0:4004-4008->4004-4008/tcp   centaurtests_centaurtestservers_1
71fccaf3fabf        stesho/selenium-remote       "sh /opt/selenium/..."   4 weeks ago         Up 4 weeks                      0.0.0.0:4444->4444/tcp                            centaurtests_selenium_1
3b61025ad614        0700d505656c                 "/bin/sh -c 'sourc..."   4 weeks ago         Up 4 weeks                                                                        silly_shirley
5244fdf06b74        stesho/centaur               "/bin/sh -c 'sourc..."   4 weeks ago         Up 4 weeks                      0.0.0.0:3004-3008->3004-3008/tcp                  objective_bassi
92f4b101ad6a        stesho/eprints3              "/bin/bash"              7 weeks ago         Up 7 weeks                      0.0.0.0:8098->8098/tcp                            eprints_pubs

After which I can login to my chosen container in an interactive shell for example:

P:\>docker exec -it objective_bassi bash

Remove Docker overlays and images

To remove all the images and their overlays:

sudo docker rmi `docker images -aq`


If there are any remaining overlays in /var/lib/docker/overlay can we just delete these?

Try to prune docker:

 sudo docker system prune -a -f

Allowing docker write access to volumes mounted on NFS

My development machine (mydev) mounts several directories which are on an NFS mounted filesystem

When I run docker I wish to be able to access and write to the NFS filesystem.

The following is a brief technique:

Inside the dockerfile:


RUN useradd -m -d /home/mydocker mydocker; echo 'usermod -u $HOST_UID mydocker' >> ~/.bash_profile

# what order does this run in?
ENTRYPOINT source /root/.bash_profile && su - mydocker && /bin/bash


If I we to run docker with:

[me@mydev]$ sudo docker -i run -e"HOST_UID=`id -u`" -v /some/nfs/drive:/var/local/drive -t /bin/bash

Then mydocker will have the same uid as the user I ran the sudo with and should be able to access the NFS drive in the same was as the native user (me).

Git: decide that the changes I have made require a new branch

Okay, so I have cloned my repo and started working on a bug fix / feature.  When I come to wanting to commit the feature figure that I need to create  a feature branch.

So I can create and switch branch by:

git checkout [feature branch name]

git status # this will show the branch that my local sandbox is not pointing to)

git commit -m"my message" [my files]

git push origin [feature branch name]

Transfer svn repos to git

(based on the experience of following:  https://www.atlassian.com/git/tutorials/migrating-convert )

required: yum install wget git java git-svn

In the svn repos get the URL of the repository:

: svn info

Get the ids of all users who have committed work to the repository:

svn log --quiet | grep "^r" | awk '{print $3}' | sort | uniq




on the 'transfer machine' setup ssh access /access to your svn repos. For example if using ssh auth create a local user

adduser

then create ssh keys for that user:

ssh-keygen -t rsa

and copy the id_rsa.pub file onto the svn server and cat  >> authorised_keys file

su -

cd ~/home/

download the transfer script:

wget https://bitbucket.org/atlassian/svn-migration-scripts/downloads/svn-migration-scripts.jar




The migrations:

Test the you have the libs and if not install / upgrade

java -jar ~/svn-migration-scripts.jar verify

if all okay

generate an authors file. I hand crafted mine:
authors.txt

jb1 = Joe Bloggs < email@somewhere.com >

Then clone the repo (we used the standard trunk / branches/ tags layout):

git svn clone --stdlayout --authors-file=authors.txt svn+ssh://svn-server.somewhere.on.int.net/usr/src/local/svn-repos/web-world/ [your git repo name]

then see what there is totidy up:

cd [your git repo name ]

java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git

java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

when happy create the repo on bitbucket (if not already)

prepare to send:

git remote add origin https://[git user]@bitbucket.org/[gitowner]/[your git repo name].git

git push -u origin --all

bish bash!

Notes:

For some reason your svn repo does not have tags or branches dirctories. The script will complain eg:

in [your git repo name]/.git/config

add to the section

[svn-remote "svn"]
    branches =

    tags =  

sails mocha: Running specific tests in mocha

To avoid running the entire test suite (when writing the tests), you can choose to only run specific tests based on a matching regular expresssion eg:

given snippet of packages file:

"scripts":{
    ...,
    "test": "PORT=9998 NODE_ENV=test mocha -R spec -b --recursive"
}

then

npm test --  --grep ManageSessions

will run any of the tests with a describe title matching *ManageSessions*

Windows docker running out of disk space

By default when installing Docker for windows the docker images file will be installed in the path configured in the Hyper-V manager, which for me happened to be in C:\Users\Public\Public Documents\Hyper-V.

To move the image file you need to:

1. Stop Docker.
2. Move the images file (MobyLinuxVM.xhdx) to the location you wish for Hyper-V to store the image
3. Open Hyper-v and in the setting change the default location to save this image
4. Uninstall and then re-install docker

voila!

locating the php.ini file

  phpinfo();
?>

Trouble running selenium script with Python Basic Auth

For pages protected by Basic Auth a little addition needs to be made to the Python script.

Whereas in a browser you can access the restricted pages by providing the username and password:

get http://username:password@somedomain.com/endpoint

you can not just put this in the url string of the Python webdriver class before you do you need to load a profile:

profile = webdriver.firefoxPorfile()

and call

profile.set_preference("network.http.phishy-userpass-length",255)

so that something like:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
profile = webdriver.FirefoxProfile()
profile.set_preference("network.http.phishy-userpass-length", 255)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("http://test0003:test@localhost/rasppi/")
driver.get("http://localhost/rasppi/")
print(driver.page_source.encode('utf-8'))
assert "Display" in driver.page_source
driver.close()

will work!

Create and manage Docker images on Windows and Docker Hub

To create a docker image on windows we need to install docker!

https://docs.docker.com/docker-for-windows/

and start the docker service.

Once this has been done prepare a directory structure, where we will organise our Dockerfiles for each image:

O:\sms67pc\docker\images\ 

Create a new directory containing a file named 'Dockerfile' for the image you wish to edit then cd to this directory edit the Docker file and once ready build the image:

# docker build -t [yourtag]  . 

to run the image interactively (cf login to the image) 

# docker run -i -t [yourtag] /bin/bash 

By logging in your can then test the further commands needed to build the image and add these to the docker file and repeat. Note, the images are not intended to be used as virtual machines and only one command can be run at the end (best to leave further commands to be run in the pipeline scripts; eg bitbucket pipeline will log you in 'interactively') 

You can start the Docker image exposing ports, to example, check the output of HTTP pages by mappng a port eg:

# docker run -i -i [yourtag] -p 8080:80 /bin/bash

If you were to start apache (inside the container) on port 80, then you should be able to view the web pages via http://localhost:8080/

Once happy upload to your docker hub account so that the images can be downloaded by tools like the bitbucket pipelines. 

# docker login 
# docker push [yourtag]

Journey splitter

Rebase modified branch

you need to put things back first:

git checkout --
git checkout --
git checkout --
git checkout --

then you can:

git rebase origin/master

List the javascript functions available in JQuery or perhaps another library

                var objs = Object.getOwnPropertyNames(jQuery);
                    for(var i in objs ){
                    console.log(objs[i]);
                }

Git on linux - a few fixes to some issues

A brief list of issues and ways to resolve when working on linux

Attempting to communicate with the server but this happens eg:

#git remote show origin

(gnome-ssh-askpass:18745): Gtk-WARNING **: cannot open display: localhost:12.0

Yep I do not have a display. Solution;

#unset SSH_ASKPASS

Attempt to push the first version of the repo but this happens:

#git push -u origin master

error: The requested URL returned error: 403 Forbidden while accessing https://github.com/S-Stephen/RECK.git/info/refs


fatal: HTTP request failed

Turns out that I cut and pasted the code from github to setup the repo which included the line:

git remote add origin https://github.com/My-Username/APPNAME.git

This does not include the username I wish to login as I therefor eneed to change this locally to be:

https://My-Username@github.com/My-Username/APPNAME.git

To do this:

# git remote set-url origin https://My-Username@github.com/My-Username/APPNAME.git

And away we go:

I did a #git add * but too much got added including my config/local.js file. 

Before the commit remove this:

# git reset config/local.js

Including emoji / unicode characters to email subject from a scipt

💣 Time to add special characters to your email subjects

I came across this problem when someone asked for emails from my web application to stand out in their inbox.  The emails were informing a user that an alarm had gone off and they would like a bomb or similar to appear in their inbox. 

After a bit of googling and finding many sites that explained how to do this manually (cut and paste characters or using particular 'clipboard' style tools).  I figured out the answer of how to embed the unicode into the message.

So given my script (a nodejs application) I followed the recipe below:

Find the character the user is interested in for example search a site like http://www.fileformat.info

Once you have found the required character locate the UTF-8 (hex string) in the case of the alarm clock (http://www.fileformat.info/info/unicode/char/23f0/index.htm): 'e28fb0'

Embed this in the subject string via: =?UTF-8?Q?=E2=8F=B0?= (ie enclose the hex in =?UTF-8?Q?= and ?=).

Send message and as long as their mail agent supports the character hey hoe!

Sails waterline ORM on an MySQL view

I am required to keep a track of the number of selections a user has made so that when they are below a particular number they can be contacted.

To do this I need to monitor a group by with having count(*) query, for which I decided to generate a view:

create view progress_view as select user.username, display_name, email, year_of_entry, count(*) as num_choices from user left join selection on user.username = selection.username where progress = 1 group by username;

unfortunately defining a regular sails model object fails to load as it complains that we are 'Trying to define a collection (progress_view) which already exists.'

to get around this and to prevent a regular query attempting to retrieve the default fields createdAt, updateAt, id when using the .find() query on the table the following was wadded to the model definition:


var Progress = {
  // Enforce model schema in the case of schemaless databases
  schema: true,

  autoPK: false,
  autoCreatedAt: false,
  autoUpdatedAt: false,
  tableName: 'progress_view',
  migrate: 'safe',

  attributes: {
    username  : { type: 'string', unique: true },
    email     : { type: 'email' }, //,  unique: true },
    display_name     : { type: 'string' }, //,  unique: true },
year_of_entry : { type: 'integer' }, //only on some will be added -> or maybe all, but generally in the inst_name?
num_choices : { type: 'integer' }, //note the minimum here is always 1! evern if the student has not chosen on (as we are identifying with left join)
  }
};

module.exports = ProgressProgress;

hurrah - we can query our view!