werk21 blog

Update in-place replacement clamav on Ubuntu

kelly — Mon, 20 Feb 2023 19:43:38 +0000

Update in-place replacement clamav on Ubuntu kelly Mon, 02/20/2023 - 20:43

For the current clamav vulnerabilites CVE-2023-20032 and CVE-2023-20052 are no updates for Ubuntu available at this time. We use the LTS Downloads 1.0.1 from clamav for a temporaray in-place replacement on Ubuntu.

We extracted from https://www.clamav.net/downloads the binaries and libs and placed them on the right place for ubuntu. Tested on Ubuntu 16.04 and 18.04. Please feedback your experience.

To install:

# Only if used with amavis
service postfix stop
service amavis stop
 
# Stop running services.
service clamav-freshclam stop
service clamav-daemon stop
 
# Remove old libraries.
rm /usr/lib/x86_64-linux-gnu/libclamav.so.*
rm /usr/lib/x86_64-linux-gnu/libfreshclam.so.*
 
 
# Download and extract LTS package.
mkdir clamav-1.0.1
cd clamav-1.0.1
wget https://www.clamav.net/downloads/production/clamav-1.0.1.linux.x86_64.deb
ar x clamav-1.0.1.linux.x86_64.deb
tar -xzf data.tar.gz
 
# Copy needed files to filesystem.
cp -a usr/local/bin usr/local/lib usr/local/sbin /usr/
 
# Run ldconfig and symlink original config.
ldconfig
ln -s /etc/clamav/clamd.conf /usr/local/etc/
ln -s /etc/clamav/freshclam.conf /usr/local/etc/
 
# Testing and starting freshclam.
freshclam --version
# ClamAV 1.0.1/26818/Mon Feb 20 09:21:31 2023
freshclam
service clamav-freshclam start
tail /var/log/syslog
 
# Starting clamav-daemon.
service clamav-daemon start
# Waiting for clamd.ctl
watch ls -l /var/run/clamav/clamd.ctl
tail -50 /var/log/syslog
 
 
# Only if used with amavis
service amavis start
tail /var/log/syslog
 
# Only if used with amavis
service postfix start
tail /var/log/mail.err
tail /var/log/mail.log

If you want to test you can send an email (or connect via telnet on port 25) and use Eicar string:

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

Reinstall Orginal packages:

service clamav-freshclam stop
service clamav-daemon stop
 
rm /usr/local/etc/clamd.conf
rm /usr/local/etc/freshclam.conf
rm /usr/lib/libclam*
rm /usr/lib/libfreshclam*
rm /usr/lib/pkgconfig/libclamav.pc
 
aptitude reinstall clamav clamav-base clamav-daemon clamav-freshclam clamdscan libclamav7 libclamav9 libclamunrar7

Systems

Server

Drupal 9: Simplenews, Swiftmailer and Attachments (via hook_mail_alter)

kelly — Tue, 30 Aug 2022 19:16:32 +0000

Drupal 9: Simplenews, Swiftmailer and Attachments (via hook_mail_alter) kelly Tue, 08/30/2022 - 21:16

In Drupal 8/9 simplenews can add attachments only from file-field (and I am not sure if this works correctly because simplenews is adding the whole file-entity to the attachments-array and swiftmailer awaits a special array).

To handle your own attachments e.g. in referenced media-entities you can use hook_mail_alter().

In this example the attached files come from the media-entities stored in "field_upload_media":

/**
 * Implements hook_mail_alter().
 */
function MYMODULE_mail_alter(&$message) {
 
  // Add Attachments from field_upload_media to simplenews.
  if (isset($message['params']['simplenews_mail'])) {
 
    // Get simplenews-mail-object.
    $mail = $message['params']['simplenews_mail'];
 
    // Get newsletter-node from mail-object.
    $node = $mail->getIssue();
 
    // Get all referenced media entities from field_upload_media.
    if ($items = $node->get('field_upload_media')->referencedEntities()) {
 
      $attachments = [];
      foreach ($items as $item) {
 
        // Get file entity from field_media_file.
        $file = $item->get('field_media_file')->entity;
 
        // Get mime-type.
        $filemime = $file->getMimeType();
 
        // Get file-uri and realpath.
        $uri = $file->getFileUri();
        $filepath = \Drupal::service('file_system')->realpath($uri);
 
        // Build array for swiftailer and add to message.
        $attachments[] = [
          'filepath' => $filepath,
          'filename' => basename($filepath),
          'filemime' => $filemime,
        ];
      }
      $message['params']['attachments'] = $attachments;
    }
  }
}

Systems

Drupal 8/9

Build xen-hypervisor 4.16 and xen-tools on Ubuntu 22.04 / 20.04 / 18.04 for PVH from source

kelly — Fri, 10 Jun 2022 13:05:36 +0000

Build xen-hypervisor 4.16 and xen-tools on Ubuntu 22.04 / 20.04 / 18.04 for PVH from source kelly Fri, 06/10/2022 - 15:05

To get latest version to protect your system or use zstd-compressed Kernels you can build your xen-hypervisor from source.

This howto based on this official howto.

Updated for Ubuntu 22.04: Boot with pygrub and xen-tools (2022-08-18)

Updates for Ubuntu 22.04: Added build dependencies (2022-06-10)

Updated to xen 4.16: Added ninja-build to build environment (2021-12-16)

Updates to xen 4.15: Added libzstd1-dev / libzstd-dev for zstd-compressed Kernels.

Xen-Hypervisor

Install requirements for build

For Ubuntu 22.04:

apt-get -qq install build-essential
apt-get -qq install bcc bin86 gawk bridge-utils iproute2 libcurl4 libcurl4-openssl-dev bzip2 kmod transfig tgif pkg-config
apt-get -qq install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended libpci-dev mercurial
apt-get -qq install make gcc libc6-dev zlib1g-dev python-all python-all-dev python3-twisted libncurses5-dev patch libvncserver-dev libsdl1.2-dev libjpeg-dev
apt-get -qq install python3-dev libglib2.0-dev
apt-get -qq install libnl-3-dev libnl-cli-3-dev libnl-genl-3-dev libnl-route-3-dev libnl-idiag-3-dev libnl-xfrm-3-dev
apt-get -qq install iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib ocamlbuild libx11-dev bison flex xz-utils libyajl-dev
apt-get -qq install gettext libpixman-1-dev libaio-dev markdown pandoc iasl cmake figlet
 
apt-get -qq install libssh2-1-dev libssh-dev libsnappy-dev
apt-get -qq install libc6-dev-i386
apt-get -qq install lzma lzma-dev liblzma-dev
apt-get -qq install libsystemd-dev
 
apt-get -qq install libzstd-dev
apt-get -qq install ninja-build

For Ubuntu 20.04

apt-get -qq install build-essential
apt-get -qq install bcc bin86 gawk bridge-utils iproute2 libcurl4 libcurl4-openssl-dev bzip2 kmod transfig tgif pkg-config
apt-get -qq install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended libpci-dev mercurial
apt-get -qq install make gcc libc6-dev zlib1g-dev python python-dev python3-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg-dev
apt-get -qq install python3-dev libglib2.0-dev
apt-get -qq install libnl-3-dev libnl-cli-3-dev libnl-genl-3-dev libnl-route-3-dev libnl-idiag-3-dev libnl-xfrm-3-dev
apt-get -qq install iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib ocamlbuild libx11-dev bison flex xz-utils libyajl-dev
apt-get -qq install gettext libpixman-1-dev libaio-dev markdown pandoc iasl cmake figlet
 
apt-get -qq install libc6-dev-i386
apt-get -qq install lzma lzma-dev liblzma-dev
apt-get -qq install libsystemd-dev
 
apt-get -qq install libzstd-dev
apt-get -qq install ninja-build

For Ubuntu 18.04:

apt-get -qq install build-essential
apt-get -qq install bcc bin86 gawk bridge-utils iproute2 libcurl4 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif
apt-get -qq install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial
apt-get -qq install make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libjpeg-dev
apt-get -qq install python3-dev
apt-get -qq install libnl-3-dev libnl-cli-3-dev libnl-genl-3-dev libnl-route-3-dev libnl-idiag-3-dev libnl-xfrm-3-dev
apt-get -qq install iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev
apt-get -qq install gettext libpixman-1-dev libaio-dev markdown pandoc iasl cmake figlet
 
apt-get -qq install libc6-dev-i386
apt-get -qq install lzma lzma-dev liblzma-dev
apt-get -qq install libsystemd-dev
 
apt-get -qq install libzstd1-dev
apt-get -qq install ninja-build

Download and prepare and make

We use the latest stable for 4.16

git clone git://xenbits.xen.org/xen.git
cd xen
git checkout origin/stable-4.16

Configure and make

Ubuntu uses systemd:

./configure --enable-systemd

For first make use:

make dist

For new make use make world (same like make clean && make dist).

Get a coffee now.

Install

The install-files are in /dist/install. And you will find an install-script in dist/install.sh. Dont use this script! It will breaks your /var/run-folder.

You can copy the files via cp -a or rsync in your system. Or you may use make debball and install the package from file (I didnt try).

Post-Install

After copying the files you will do some changes in your system:

Reload dynamic libraries:

/sbin/ldconfig

Grub-Settings:

mkdir -p /etc/default/grub.d

Create /etc/default/grub.d/xen.cfg:

vi /etc/default/grub.d/xen.cfg

GRUB_DISABLE_OS_PROBER=true
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=2560M,max:2560M"
GRUB_DEFAULT="Ubuntu GNU/Linux, mit Xen-Hypervisor"

Be careful! The last settings depends on your system-language! For english use: "Ubuntu GNU/Linux, with Xen hypervisor" - Ubuntu 18.04 does not use the german translation!

You have to update grub:

update-grub

Enable systemd-files:

In the current version is a bug and xendomains.service and /etc/init.d/xendomains are created both. To use systemd you have to remove /etc/init.d/xendomains

rm /etc/init.d/xendomains

systemctl enable xen-qemu-dom0-disk-backend.service
systemctl enable xen-init-dom0.service
systemctl enable xenconsoled.service
systemctl enable xenstored.service
systemctl enable xendomains.service

Optional you can enable xen-watchdog.service

systemctl enable xen-watchdog.service

Configure xen

Configure xen for your needs.

xen-tools

For creating new VMs you can use xen-tools. Unfortunately you cannot use the ubuntu-package because it requires xen-hypervisor. So we build from source.

Install requirements for build

apt install lvm2 debootstrap libconfig-inifiles-perl libdata-validate-domain-perl libdata-validate-ip-perl libdata-validate-uri-perl libfile-slurp-perl libfile-which-perl libsort-versions-perl libterm-ui-perl libtext-template-perl openssh-client perl debian-archive-keyring rinse libtest-notabs-perl

Download and prepare and make

git clone https://github.com/xen-tools/xen-tools.git
cd xen-tools
make install

If you want only the install-files (e.g. for build you own package), you can edit the Makefile and manually set the variable DESTDIR before make install.

Adaptions for Ubuntu newer versions incl. 22.02

For newer versions the symlink to artful is missing. Just create symlinks and add the distri to distributions.conf

cd usr/share/xen-tools/
ln -s artful.d bionic.d
ln -s artful.d focal.d
ln -s artful.d jammy.d

echo "bionic      = ubuntu     pygrub" >> etc/xen-tools/distributions.conf
echo "focal       = ubuntu     pygrub" >> etc/xen-tools/distributions.conf
echo "jammy       = ubuntu     pygrub" >> etc/xen-tools/distributions.conf

Ubuntu 22.04 and Grub

Ubuntu 22.04 contains no Grub-1 anymore. So we just add a /boot/grub/menu.lst manually so pygrub can boot.

cat << EO2 > usr/share/xen-tools/artful.d/99-pseudogrub
 
#!/bin/sh
#
#  This script enable boot via pygrub without grub-1.
#

 
prefix=$1
 
#
#  Source our common functions
#
if [ -e /usr/share/xen-tools/common.sh ]; then
    . /usr/share/xen-tools/common.sh
else
    . ./hooks/common.sh
fi
 
#
# Log our start
#
logMessage Script $0 starting
 
mkdir -p ${prefix}/boot/grub
cat << EOF > ${prefix}/boot/grub/menu.lst
 
default         0
timeout         2
 
title           Ubuntu 22.04 LTS
root            (hd0,0)
kernel          /boot/vmlinuz root=/dev/xvda2 ro elevator=noop
initrd          /boot/initrd.img
 
title           Ubuntu 22.04 LTS (Single-User)
root            (hd0,0)
kernel          /boot/vmlinuz root=/dev/xvda2 ro single elevator=noop
initrd          /boot/initrd.img
 
title           Ubuntu 22.04 LTS (Old)
root            (hd0,0)
kernel          /boot/vmlinuz.old root=/dev/xvda2 ro elevator=noop
initrd          /boot/initrd.img.old
 
title           Ubuntu 22.04 LTS (Old, Single-User)
root            (hd0,0)
kernel          /boot/vmlinuz.old root=/dev/xvda2 ro single elevator=noop
initrd          /boot/initrd.img.old
 
EOF
 
#
# Log our finish
#
logMessage Script $0 finished
 
EO2
 
chmod +x usr/share/xen-tools/artful.d/99-pseudogrub

Configure xen-tools

Configure xen-tools for your needs.

Reboot

After reboot you have your Ubuntu on xen-hypervisor.

Test:

xentop

Settings for PVH

For PVH you have to use Linux 4.11 or newer (e.g. HWE-Kernel) in guest:

apt install linux-virtual-hwe-16.04 linux-tools-virtual-hwe-16.04

Then you can use:

bootloader = '/usr/local/bin/pygrub'
type="pvh"

If you have an old kernel in your guest-system, you have to boot from dom0-kernel with this settings:

type="pvh"
kernel='/vmlinuz'
ramdisk="/initrd.img"

Systems

Server

Tags

How to "fix" 403 access denied exceptions for Drupal 8 Views translation routes

patrick — Thu, 06 Dec 2018 07:50:41 +0000

How to "fix" 403 access denied exceptions for Drupal 8 Views translation routes patrick Thu, 12/06/2018 - 08:50

Today I worked on a project where the customer had asked us to add a missing translation for the label of an exposed filter of a view. The view configuration was originally imported upon installation of a contributed module, that uses that view. I thought that this would be an easy task quickly done: We have configuration translation enabled on that system so all views have a "Translate" operation that you can use to translate the view. Not in this case, of course ;).

I was logged in as the magic super user 1 with the magic administrator role assigned to it, but I still got a 403 access denied exception each time I tried to translate this particular view. Other views that we had created manually or that were provided by other contribued modules worked fine. But not this one. What the ****?

After searching the core issue queue for a while to no avail, I ended up to comment out the "throw AccessDeniedException" in the AccessAwareRouter, just out of curiosity. After all this was a dev system, so what could possibly go wrong? And then I instantly realized what the problem was. The list of translations was displayed and I saw that the "original translation" was "language undefined" or "und". Looking at the configuration object more closely confirmed, that the language was "und". In hindsight, it was kind of logical that I couldn't translate that one, right?

Of course I didn't make the foolish attempt to create an English translation for the view via UI. Since Views doesn't have a UI to change the language of a view like for example Content entities have, I decided to just export the configuration using Drush, find the YAML file for the configuration object in question, edit it so that the language was no longer "und", but "en", and import the configuration again using Drush. Now I was able to translate the view using the UI without my quick modification of the AccessAwareRouter.

Yet another lesson learned.

Systems

Drupal 8/9

Tags

drush Views Translation Config Translation yml AccessAwareRouter AccessDeniedException Exception

Open multiple key database files upon KeePass2 startup

patrick — Mon, 24 Sep 2018 06:19:16 +0000

Open multiple key database files upon KeePass2 startup patrick Mon, 09/24/2018 - 08:19

KeePass2 has a setting to open the most recently opened key database upon start-up, but what if you want to open multiple key database files? One way to do this is to add a trigger that will open the files on start-up.

Go to Tools > Options > Advanced and make sure that Remember and automatically open last used database on startup is unchecked.
Go to Tools > Triggers and click on the Add button.
On the Properties tab, specify a convenient name to your liking.
On the Events tab, click the Add button and add an Application started and ready event.
On the Actions tab, cklick the Add button and add an Open database file action. As File/URL setting, specify the full path to your key database file. If you use a password to open your database, it is suggested to leave the Password setting empty as it will be stored in unencrypted form. You will then be prompted to enter the password upon start-up as usual.
Add additional Open database file actions for each database you want to open.

Systems

Tags

KeePass2

Date (range) fields and Entity Query [Update]

patrick — Mon, 05 Feb 2018 07:09:16 +0000

Date (range) fields and Entity Query [Update] patrick Mon, 02/05/2018 - 08:09

Ever wanted to add a condition on the value of a date field on an entity query in Drupal 8? Here is how it works reliably.

For Drupal 8.5.x and later:

use Drupal\Core\Datetime\DrupalDateTime;
use Drupal\datetime\Plugin\Field\FieldType\DateTimeItemInterface;
 
// Get a date string suitable for use with entity query.
$date = new DrupalDateTime();
$date->setTimezone(new \DateTimeZone(DateTimeItemInterface::STORAGE_TIMEZONE));
$date = $date->format(DateTimeItemInterface::DATETIME_STORAGE_FORMAT);
 
// Set the condition.
$query->condition('field_date.value', $date, '>');

For Drupal 8.4.x and earlier:

use Drupal\Core\Datetime\DrupalDateTime;
 
// Get a date string suitable for use with entity query.
$date = new DrupalDateTime();
$date->setTimezone(new \DateTimeZone(DATETIME_STORAGE_TIMEZONE));
$date = $date->format(DATETIME_DATETIME_STORAGE_FORMAT);
 
// Set the condition.
$query->condition('field_date.value', $date, '>');

The trick is to use a DrupalDateTime object, on which we set the proper time zone used by the storage (provided by the constant DATETIME_STORAGE_TIMEZONE) and format that in the proper date format used by the storage (provided by the constant DATETIME_DATETIME_STORAGE_FORMAT). That way, we'll get reliable results. In the example above I just wanted to filter by the current time, but you can pass a date string to the constructor of DrupalDateTime as well, if you want something else.

Please note: Starting with Drupal 8.5.x, the constants mentioned above have been deprecated in favor of constants on a new DateTimeItemInterface. They will be removed in Drupal 9.x.

You can also get these objects directly from a date field's value by using getDateTime():

foreach ($node->field_date_range as $item) {
  foreach ($item as $index => $value) {
    $date = $value->getDateTime();
  }
}

If you're using experimental date range fields, another thing you need to know is, that the end date is stored in an end_value column.

Here's a more complete example querying for event nodes in the future:

use Drupal\Core\Datetime\DrupalDateTime;
use Drupal\Core\Language\LanguageInterface;
 
// @note: In Drupal 8.5.x and later also include the following line:
use Drupal\datetime\Plugin\Field\FieldType\DateTimeItemInterface;
 
$storage = \Drupal::service('entity_type.manager')->getStorage('node');
$query = $storage->getQuery()
  ->condition('type', 'event')
  ->condition('status', 1)
  ->sort('field_date_range.value', 'ASC')
  ->sort('field_date_range.end_value', 'ASC')
  ->sort('created', 'ASC');
 
$query->condition('langcode', [
  \Drupal::service('language_manager')->getCurrentLanguage()->getId(),
  LanguageInterface::LANGCODE_NOT_APPLICABLE,
  LanguageInterface::LANGCODE_NOT_SPECIFIED,
], 'IN');
 
// @note BEGIN: Code for Drupal 8.5.x and later: 
$date = new DrupalDateTime(); 
$date->setTimezone(new \DateTimeZone(DateTimeItemInterface::STORAGE_TIMEZONE)); 
$date = $date->format(DateTimeItemInterface::DATETIME_STORAGE_FORMAT);
// @node END: Code for Drupal 8.5.x and later.
 
// @note BEGIN: Code for Drupal 8.4.x and earlier:
$date = new DrupalDateTime();
$date->setTimezone(new \DateTimeZone(DATETIME_STORAGE_TIMEZONE));
$date = $date->format(DATETIME_DATETIME_STORAGE_FORMAT);
// @node END: Code for Drupal 8.4.x and earlier.
 
$and = $query->andConditionGroup()
  ->condition('field_date_range.value', $date, '>')
  ->notExists('field_date_range.end_value');
 
$or = $query->orConditionGroup()
  ->condition($and)
  ->condition('field_date_range.end_value', $date, '>');
 
$query->condition($or);
 
$ids = $query->execute();
$nodes = $storage->loadMultiple($ids);

Note, that there's currently an issue in Drupal Core with nested condition groups in entity query, which are used in the example above.

Update: This post has been updated to reflect changes in Drupal 8.5.x (see Global constants in datetime.module are deprecated and DateTimeItemInterface has been introduced).

Systems

Drupal 8/9

Tags

date date range LanguageManager getCurrentLanguage LanguageInterface DrupalDateTime entity query EntityTypeManager DateTimeItemInterface

Install pfsense 2.4.x on xen (Ubuntu 16.04)

kelly — Sun, 17 Dec 2017 16:43:23 +0000

Install pfsense 2.4.x on xen (Ubuntu 16.04) kelly Sun, 12/17/2017 - 17:43

pfsense 2.4.x is not available as nano-image, so you have to install pfsense as xen-Guest via vnc.

Download pfsense

Download pfsense from https://www.pfsense.org/download/ (AMD64, CD Image (ISO) Installer)

Configure Xen

You have to create a LV or another device to install.

builder     = 'hvm'
vcpus       = '4'
memory      = '2048'
 
disk        = [
                  'phy:/dev/xenvg/pfsense-disk,hda,w',
                  'file:/root/pfSense-CE-2.4.2-RELEASE-amd64.iso,hdc:cdrom,r'
              ]
 
serial      = 'pty'
 
vnc         = 1
vnclisten   = '0.0.0.0'
 
boot        = 'd'
 
name        = 'pfsense'
 
vif         = [
                'ip=xxx.xxx.xxx.xxx,bridge=xenbr0',
                'ip=yyy.yyy.yyy.yyy,bridge=xenbr1',
                'ip=zzz.zzz.zzz.zzz,bridge=xenbr2',
              ]

Installing pfsense

Create Guest (xl create) and connect with your vncviewer (https://www.realvnc.com/en/connect/download/viewer/)

(Remark: We had to add internal route / tunnel to access webgui via browser)

After installing pfsense

After installing remove lines for booting and vnc:

builder     = 'hvm'
vcpus       = '4'
memory      = '2048'
 
disk        = [
                  'phy:/dev/xenvg/pfsense-disk,hda,w'
              ]
 
serial      = 'pty'
 
name        = 'pfsense'
 
vif         = [
                'ip=xxx.xxx.xxx.xxx,bridge=xenbr0',
                'ip=yyy.yyy.yyy.yyy,bridge=xenbr1',
                'ip=zzz.zzz.zzz.zzz,bridge=xenbr2',
              ]

Important Network-settings

Under https://<your_host>/system_advanced_network.php you have to check following options (check to disable):

Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading

Systems

Server

Tags

xen ubuntu pfsense

Dealing with DEPTH TOO GREAT in Kint

patrick — Wed, 13 Dec 2017 07:58:54 +0000

Dealing with *DEPTH TOO GREAT* in Kint patrick Wed, 12/13/2017 - 08:58

If you use Devel's dpm() with Kint for debugging in Drupal 8, chances are you already encountered the message *DEPTH TOO GREAT* when clicking through a deeply nested object.

There is a reason for that: Performance. Always rendering all levels of a deeply nested object might result in either your server or your browser to run out of resources and fail. That's why there is a maximum number of levels to show. It's usually at 7 levels.

If you need to inspect something that is nested further below, the best thing to do is to limit the dpm() to a more explicit part of the nested variable. But sometimes you can't or don't want to do that. In those cases, you can increase the maximum levels by placing

kint_require();
\Kint::$maxLevels = 8;

somewhere above your dpm() call. $maxLevels is of course the maximum number of levels you want to see. If you know what you're doing, you could even use

\Kint::$maxLevels = 0;

to remove the limit entirely. Use at your own risk.

Systems

Drupal 8/9

Tags

Devel Kint dpm

Change source id mapping of a Drupal 8 migration

patrick — Wed, 13 Dec 2017 05:57:34 +0000

Change source id mapping of a Drupal 8 migration patrick Wed, 12/13/2017 - 06:57

Drupal Core's Migrate API maintains an internal table mapping unique source ids to destination ids. This is usually done via a database table that holds the field values that together uniquely identify a source row and the matching destination id of the entity in Drupal. The field names and types that uniquely identify a source row are provided by the source plugin of a migration. In most cases, you just have some kind of an id column, but migrate supports any number of fields for the primary key of the source row.

Recently I was dealing with a migration that was using the Url source plugin provided by Migrate Plus to import files from an XML source. The source plugins provided by Migrate Plus allow you to specify the ids returned by MigrateSourceInterface::getIds() in the YML file of your migrations. Originally I used a simple id map like this:

source:
  ids:
    id:
      type: integer

Unfortunately, I later noticed that I needed to add the file name as a second key to the id map, so I changed the YML file to something like this:

  ids:
    id:
      type: integer
    file_name:
      type: string

So far so good. When I wanted to update the migration, I was greeted by this error message:

Migration failed with source plugin exception: SQLSTATE[42S22]: Column not found: 1054 Unknown column "sourceid2" in
"field list": INSERT INTO {migrate_map_example_file} (source_ids_hash, sourceid1, sourceid2, source_row_status,
rollback_action, hash, last_imported) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2,
:db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array
(
    [:db_insert_placeholder_0] => a8d0cbb8ca5f48c73993244fd275342d84b92a2eb33300b0852ef560f70aa55d
    [:db_insert_placeholder_1] => 10817
    [:db_insert_placeholder_2] =>
    [:db_insert_placeholder_3] => 2
    [:db_insert_placeholder_4] => 0
    [:db_insert_placeholder_5] =>
    [:db_insert_placeholder_6] => 1513082182
)

Solving this is actually way easier than I thought. Just drop the migrate mapping table for your migration from the SQL database:

DROP TABLE migrate_map_example_file;

If you are less adventurous, you might just want to rename the table initially:

RENAME TABLE migrate_map_example_file TO migrate_map_example_file_backup;

As soon as you rerun your migration, migrate will recreate the table for you, this time using the new id mapping. Voilá.

Systems

Drupal 8/9

Tags

migrate migrate_plus source Url MigrateSourceInterface getIds

Illegal mix of collations exception caused by Drupal 8 Autocomplete with Entity Reference View

patrick — Fri, 08 Sep 2017 09:48:14 +0000

Illegal mix of collations exception caused by Drupal 8 Autocomplete with Entity Reference View patrick Fri, 09/08/2017 - 11:48

Looking at a Recent log messages report of one of our Drupal 8 instaces, I came across this error message:

Drupal\Core\Database\DatabaseExceptionWrapper: Exception in Media Autocomplete[media_autocomplete]: SQLSTATE[HY000]: General error: 1267 Illegal mix of collations (ascii_general_ci,IMPLICIT) and (utf8mb4_general_ci,COERCIBLE) for operation 'like': SELECT media_field_data.langcode AS media_field_data_langcode, media_field_data.mid AS mid, media_field_data.name AS media_field_data_name, media_field_data.bundle AS media_field_data_bundle FROM {media_field_data} media_field_data WHERE ((media_field_data.bundle = :db_condition_placeholder_0) AND ((media_field_data.name LIKE :db_condition_placeholder_1 ESCAPE '\\') OR (media_field_data.bundle LIKE :db_condition_placeholder_2 ESCAPE '\\'))) AND ((media_field_data.status = :db_condition_placeholder_3) AND (media_field_data.langcode IN (:db_condition_placeholder_4, :db_condition_placeholder_5, :db_condition_placeholder_6))) LIMIT 10 OFFSET 0; Array ( [:db_condition_placeholder_0] => image [:db_condition_placeholder_1] => %gläsernes labor% [:db_condition_placeholder_2] => %gläsernes labor% [:db_condition_placeholder_3] => 1 [:db_condition_placeholder_4] => de [:db_condition_placeholder_5] => und [:db_condition_placeholder_6] => zxx ) in Drupal\views\Plugin\views\query\Sql->execute() (line 1488 of core/modules/views/src/Plugin/views/query/Sql.php).

Clearly, an editor was trying to use the autocomplete functionality of an entity reference field searching for media entities matching the search string "gläsernes labor" (in English: transparent laboratory). This would actually happen for any search string entered into the autocomplete that didn't contain ascii characters only, so in this case the "ä" in "gläsernes" was causing the exception. Looking closer at the query, we can see that the MySQL table is "media_field_data" and the search string is in two placeholders: "db_condition_placeholder_1" and "db_condition_placeholder_2". These are used to check for matches in "media_field_data.name" or "media_field_data.bundle" columns. So I asked the database for the collation of these fields:

MariaDB [my_drupal_8_database]> SELECT TABLE_NAME, COLUMN_NAME, COLLATION_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA='my_drupal_8_database' AND TABLE_NAME='media_field_data' AND COLUMN_NAME IN ('name', 'bundle');
+------------------+-------------+--------------------+
| TABLE_NAME       | COLUMN_NAME | COLLATION_NAME     |
+------------------+-------------+--------------------+
| media_field_data | bundle      | ascii_general_ci   |
| media_field_data | name        | utf8mb4_general_ci |
+------------------+-------------+--------------------+

Tada! The bundle column of the table uses ascii collation. For our autocomplete search to work, those two columns would need to have the same collation or we would need to change the query to specify the collation to use for each like condition separately. Another option would be to just query the name field or the bundle field.

But why was the bundle field searched in the first place? We set up an entity reference view for this entity reference field (after applying a patch, mind you ;). We did this because we wanted to show the bundle and some other stuff along the label of the entity in the results returned by the autocomplete. When we set up the view we selected "Name" and "Bundle" for the "Search fields" setting of the "Entity reference list" style plugin.

Our client didn't really care that much about searching the bundle field, so I just removed it from the search fields. And voilá, you could enter strings with umlauts into the autocomplete again.

Systems

Drupal 8/9

Tags

Autocomplete Views Entity Reference Field mysql Exception